Skip to main content

Full text of "Transactions American Math Society"

See other formats


VOLUME 355 

NUMBER 7 

JULY ( 
WHOLE NUMBER  

RANSACTION 
OF THE  

AMERICAN MATHEMATICAL SOCIET 

EDITED BY 
Dan Abramovich 
Peter W. Bates 
Patricia E. Bauman 
William Beckner, Managing Editor 
Krzysztof Burdzy 
Tobias Colding 
Harold G. Diamond 
Sergey Fomin 
Lisa C. Jeffrey 
Alexander Nagel 
D. H. Phong 
Stewart Priddy 
Theodore Slaman 
Karen E. Smith 
Robert J. Stanton 
Daniel Tataru 
Abigail Thompson 
Robert F. Williams 

PROVIDENCE, RHODE ISLAND USA 
ISSN 0002-9947 

Available electrol,i," 



Transactions of the American Mathematical Society 
This journal is devoted entirely to research in pure and applied mathematics. 
Submission information. Sec Information for Authors at the end of this issue. 
Publisher Item Identifier. The Publisher Item Identifier (PII) appears at the top 
of the first page of each article published in this journal. This alphanumeric string of 
characters uniquely identifies each article and can be used for future cataloging, searching, 
and electronic retrieval. 
Postings to the AMS website. Articles are posted to the AMS website individuallv 
after proof is returued frein authors and before appearing in an issue. 
Subscription information. Transactions of the American Mathematical Socety is 
published nlonthly. Beginning in January 1996 Transactions is accessible from -.ams. 
org/publ±cations/. Subscriptiou prices for Volume 355 (2003) are as follows: for paper 
delivery, $1490 list, $1192 institutional member, $1341 corporate member; for electronic 
delivery, $1341 list, $1073 iustitutional llleml»er, $1207 corporate nlenlber. Upon request, 
subscribers to paper delivery of tl,is journal are also entitled to receive electronic delivery. 
If ordering the paper version, add $39 for surface delivery outside the United States and 
India: $511 te lndia. Expedited delivery te destillatiols in North Alnerica is $48; elsewhere 
$144. For paper dclivery a late charge of 10% of the subscription price will be inlposed 
up(m orders receivcd frein llOlllll,llll{'rs after Jalmary 1 of thc subscription year. 
Back nUlllber information. For back issues sec . ams. org/bookstore. 
Subscriptious aud orders shouht be addressed te the Anlerican Mathenlatical Society, 
P.O. Box 845904. Boston. MA 02284-5904 USA. Ail orders must be accorapanied by pay- 
ment. Other correspondence should be addressed te 201 Charles Street, Providence, RI 
1129114-2294 l'SA. 
Copying and reprinting. Material in this journal rnay be reproduced by any means 
for educational and scientific purposes without fee or permission with the exception of re- 
production by services that collect fees for delivery of documents and provided that the cus- 
tomary acknowledgment of the source is given. This consent does hot extend to other kinds 
of copying for general distribution, for advertising or promotional purposes, or for resale. 
[equests for permission for commercial use of material should be addressed to the Acqui- 
sitions Department. American Mathematical Society, 201 Charles Street. Providence, RI 
02904-2294 USA. Requests can also be lnade by e-mail to reprint-perm±ssionams.org. 
Excluded from these provisions is material in articles for which the author holds copy- 
right. In such cases, requests for permission to use or reprint should be addressed directly 
to the author(s). (Copyright ownership is indicated in the notice in the lower right-hand 
corner of the first page of each article.) 

Transactions o] the Amemcan Mathematical Society is published monthly by the Amer- 
ican Mathematical Society at 201 Charles Street, Providence, RI 02904-2294 USA. Peri- 
odicals postage is paid at Providence. Rhode Island. Postmaster: Send address changes to 
Transactions, American Mathematical Society, 201 Charles Street, Providence, RI 02904- 
2294 USA. 
@ 2003 by the American Mathematical Society. All rights reserved. 
This journal is indexed in Mathematical Reviews, Zentralblatt MA TH, Science Citation 
Index @, Science Citation IndexrM-Expanded, ISI Alerting Services TM, CompuMath 
Citation Index @, and Current Contents® / Physical, Chemical & Earth Sciences. 
Printed in the United States of America. 
( The paper used in this journal is acid-free and falls within 
the guidelines established to ensure permanence and durability. 

10 9 8 7 6 5 4 3 2 1 a a7 aa a a a 



TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY 
CONTENTS 
Vol. 355, No. 7 Whole No. 818 July 2003 

Borislav Karaivanov, Pencho Petrushev, and Robert C. Sharpley, 
Algorithms for nonlinear piecewise polynomial approximation: Theoret- 
ical aspects .......................................................... 2585 
JSrg Brendle, The ahnost-disjointness nmnber may have countable 
cofinality ............................................................ 2633 
Alina Carlnen Cojocaru, Cyclicity of Chi elliptic curves modulo p ..... 2651 
Tonghai Yang, Taylor expansion of an Eisenstein stries .................. 2663 
Eric Freelnan, Systems of diagonal Diot)hantiim inequalities ............. 2675 
Francisco Javier Callego and Bangere P. Purnaprajna, On the 
canonical rings of covers of surfaces of minimal degree ............... 2715 
H. H. Brungs and N. I. Dubrovin, A classification and examples of tank 
one chain domains ................................................... 2733 
Donald W. Barnes, Oll the si)ectral sequence constructors of Guichardet 
and Stefan ........................................................... 2755 
Steven Lillywhite, Formality in an cquivariant setting ................... 27ïl 
Neil Hindman, Dona Strauss, and Yevhen Zelenyuk, Large rectangular 
semigroul)s in Stone-Cech compactifications .......................... 2795 
Takehiko Yalnanouchi, Galois groups of quantum group actions and 
regularity of fixed-point algebras ..................................... 2813 
Boo Riln Choe, Hyungwoon Koo, and Wayne Slnith, Composition 
operators acting on holomorphic Sobolev spaces ..................... 2829 
B. Jakubczyk and M. Zhitolnirskii, Distributions of cora.nk 1 and their 
characteristic vector fields ........................................... 2857 
E. Boeckx, When are the tangent sphcre bundles of a 1Rielnalmian ma.lfifold 
reducible? ........................................................... 2885 
Henri Colnlnan, Criteria for large deviations ........................... 2905 
Seung Jun Chang, Jae (Iii Choi, and David Skoug, Integration by parts 
fornmlas involving generalized Fourier-Feylnan transforms on flmction 
space ................................................................ 2925 
Miehiko Yuri, Thermodynamic formalism for countable to one Markov 
systems .............................................................. 2949 
D. G. De Figueiredo and Y. H. Ding, Strongly indefinite flmctionals 
and lnultiple solutions of elliptic systelns ............................. 2973 
F. Rousset, Stability of sinall alnplitude boundary layers for lnixed 
hyperbolic-parabolic systenls ........................................ 2991 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETh 
oluine 355, Number 7, Pages 2585 2631 
S 0002-9947(03)03141-6 
Article electronically published on March 19, 2003 

ALGORITHMS FOR NONLINEAR 
PIECEWISE POLYNOMIAL APPROXIMATION: 
THEORETICAL ASPECTS 

BORISLAV KARAIVANOV. PENCHe PETRUSHEV, AND ROBERT C. SHARPLEY 

ABSTRACT. In this article algorithms are developed for nonlinear n-terre 
Courant element approximation of functions in Lp (0 < p <_ oo) on bounded 
polygonal domains in ll 2. Redundant collections of Courant elements, which 
are generated by multilevel nested triangulations allowing arbitrarilv sharp 
angles, are investigated. Scalable algorithms are derived for nonlinear approx- 
imation which both capture the rate of the best approximation and provide 
the basis for numerical inlplenlentation. Sinlple thrcsholding criteria enable 
approximation of a target ftlnction f te optinlal|y high asynlptotic rates which 
are deternlined and automatically achieved by the inherell| Sllloothlless of f. 
The algorithms provide direct approxinlation estimates and permit utilization 
of the general .lackson-Bernstein machinery te characterize n-terre Courant 
element approximation in terres of a scale of smoothness spaces (B-spaces) 
which govern the approximation rates. 

1. INTRODUCTION 

Highly detailed Digital Terrain Elevation Data (DTED) and associated ilnagery 
are new becolning widely availal)le for lnOSt of the earth's surface. However, al- 
gorithms for effective approximatiol of data of this type are net yet available. A 
primary lnotivation for this work is the developlnent of effective algorithms for 
nonlinear piecewise polynolnial approximation of DTED maps frein a redundant 
hierarchial system over (possibly) irregular triangulations which are constructive 
in nature. Application of the ideas and theory flOln [4] te the resulting framework 
will permit optilnal entropy tree encoding of the elevation data, elmble progressive 
view-dependent refinements which lnay be focused te user-localized regions, and 
permit the registration of similarlv encoded ilnage textures te the surface (see [10], 
[4] for lnore details). 
Our philosophy is that dependable practical approximation procedures can be 
built only upon a solid theoretical basis. Accol'dingly, we have two primary goals 
in this paper. The first is te better understand nonlinear piecewise polynolnial 

Received by the editors May 2, 2002. 
2000 Mathematics Subject Classification. Primary 41A17, 41A25, 65D18; Secondary 65D07. 
42B35. 
Key words and phrases. Nested irregular triangulations, redundant representations, nonlinear 
n-terre approximation. Courant elements, Jackson and Bernstein estimates. 
The second and third authors were suppoïted in part by Grant NSF #DMS-0079549 and ONF( 
N00014-01-1-0515. 
AI1 three authors were supported in part by ONR grant N00014-00-l-04ï0. 

2003 American Mathematical Societ. 

2585 



2586 B. KAIAIVANOV, P. PETRUSHEV, AND IR. C. SHAIPLEY 

approximation, in particular, to UlMerstand the natme of the global smoothness 
conditions (spa.ces) which govern the rate of approximation. The second goal is to 
develop or refine existing constructive approximation methods for nonlinear approx- 
ilnation which capture the rate of the best approximation and tan be implemented 
effectivelv in practice. 
This paper addresses nonlinear n-terre approximation by Courant elements gen- 
era.ted by multilevel nestcd triangulations, glore precisely, for a giwn bounded 
polygonal domain E C N 2. let (oE)0 be a sequence of triangnlations such that 
each level OE is a triangulation of E consisting of closcd triangles with disjoint 
interiors and a refillellll,ltt of tho in'crions level OEn-1- VP impose some lltild natu- 
rai conditions on the triangnlations in order to prevent possible deterioration, but 
out results arc valid for fairlv generM triangula.tions with sharp angles.  define 
Z := U,20 OE,- Each such nmltilevel triangulation Z gencrates a ladder of spaces 
0 C 81 C • • • consisting of piecewise lincar flmctions, where ,, (m  O) is spanned 
Iv ail Courant el'nelts ç0 SUl)ported on oeils 0 at the m-th level . 
Utilizing these primal elemems, we cmsider mmlinear approximation bv n-terre 
piecewise liner fmwtims of the form N = j= aç, where  mav conte from 
diren levels mM lwatims. ()ur first gml is to «haracterize the approximation 
spa«es «onsisting f Ml fimctions wilh a giveu rate of aplroxinmtion. For approxi- 
matière in L v, p < , lhis is donc lu [1 Il, wh«re a collection of smoothness spaces 
(called B-spaees) was introdu«ed and ntilized, ht this 1)aper, we develop this theory 
in the more complicated case of approximation in the uniform norm (p = oe). Out 
progrmn consists f the following steps. First. in order to qumttify the approxima- 
tion process, we dvvelop a collection of smoothness spaces B (T) which depend on 
T and will govern the best approximation. Second. we prove compmfion Jackson 
and Bernstein estimates, mtd. third, we clmracterize the approximation spaces bv 
interpolation spa('e methods. 
(htr second and prinmry goM is, by using the B-spa«es mM the related tech- 
niques, to develop (or refine) Mgorithms for nonlinear n-terre Courront element 
al»proximation so that the new Mgorithms are capMfle of achieving the rate of the 
best approximation. In the present paper, we develop three such algorithms for 
-term Courant element approxintation in L v, which we call "'threshold'" (p < ), 
"'trim and cut'" (0 < p  ), mM "'Imsh the error'" (p = ) algorithnts. 
The first step of ea«h of th«se algorithms is a d««omposition step. We denote 
by O the set of all cells (supports of Courant elements) generated bv 
(ç)eo is obviouslv redmMant mtd, therefore, every fmtction f lins infinitely many 
representations of the form 
(-)  =  (f)ç- 
It is crucial to have a suciently ecient (sparse) initial representation of the fimc- 
tion f çhat is being approximated. In our case. this means that the representation 
(1.1) of f should allow a reMization of the corresponding B-notre [[f[[B(OE). Thus 
the problem of obtaining an ecienç iniçial representation of the fimctions is tightly 
related ço çhe development of the -spaces.  achieve such eciencv bv using 
good projecçors into the spaces , m = 0, 1 ..... 
For compleçeness and comparison, we first consider çhe naturM "threshold'" al- 
gorithm for -çerm Courront elemenç approxinmçion, which is valid only in 
0 < p < .. This algorithm simply takes the largest (in Lp) n-terres from (1.1). 



NONLINEAR PIECEW'ISE POLYNOMIAL APP1ROXINIATION 2587 

Usiug the results ri'oto [11], if is easy fo show that the "threshold'" atgorithm cap- 
turcs the rate of the best n-terre Courant element approximation in Lp (p < ). 
The second algorithm, which we call "trim aud cut", originates from the proof 
of the Jackson estimate in [7] and uses the following idca. First, we partition  
through a coloring into a fmnily of disjoint trees " (with respect to the inclusion 
relation):  := U.= Second. we "trim'" each tree by removing cells 0 G " 
corresponding to iusiguificant small terres aoço from (1.1), located near the t, ips of 
the branches. Third, we divide ("cut") the remaining parts of each tree " into 
sections of small "energy". Finally, we rewrite the significant part of each section as 
a linear combination of a slnall number of Courant elements. The resulting terres 
determine the final approxilnmt. b shall show that "'triln and cut" is capable of 
achieving the rate of the best approximation iii Lp (0 < p  ). 
Pivotal in our development is the "push the error" algorithm, the nmne of which 
was coined bv Nira Dvn. The idea for this algorithm appears in [5] and mav be 
roughly described in LOe as follows. For a fixed e > 0. we "'Imsh the error" with 
«, starting ffoto the coarsest level (-)0 and proceeding to finer levels. Nalnely, we 
denote by A0 the set of ail 0  (-)0 such that laol > e (11011 = 1) and define 
0 := 0e aoço. Theu we rewrite ail remaining terres aoço at the next level and 
add the resultiug terlns to the existing terres aoço, 0  (-)1- We denote the uew 
terlns by doço, 0  Or, and select in A1 all 0  (-)1 SllCh that Idol > e. xx contime 
pushing the error in this wav to the finer levels in the lepreseutation of f. Finallv. 
we define our approxilnant bv A := k0 Aj. Thus terres doço with Idol  ¢ are 
discarded only at a very fine level, and hence the error (in L) is  e. 
Of course, this haire "push the error" algorithm calmot achieve the rate of the 
best. approximation. However,  we shall show iii 3.3 and 5. after solne substan- 
tial improvelnents, the algorithm is capable of achieving the rate of convergence of 
the best n-terre Com'ant element aI)proximation in the uniform norm. 
A focal point of our developlnent is the characterization of the approximation 
spaces generated by the best n-terre Courant element approximation in L and 
the characterization of certain approximatiou spaces sociated with the three al- 
gorithms developed, which show that they capture the rate of convergence of the 
best approximation. 
The outline of the paper is as follows. In 2. We collect all facts needed regarding 
multilevel triangulations, local approximation, quasi-interpolmts, and B-spaces. In 
3. we develop and explore the three algorithms for nonlinear n-terln Courant ele- 
ment approximation: "'t hreshold" algorit hm (in 3.1), "t riln and eut'" algorithm (in 
3.2), and "push the error" algorithln (in 3.3). Section 4 is devoted to establishing 
3ackson and Bernstein inequalities in order to study best n-terre Courant element 
approximation. In 5, we show that the three algorithlns capture the rate of the 
best n-terre Courant elelnent approximation and identifv the associated approxi- 
mation spaces  B-spaces. In 6, we discuss some of the main issues of nonlinear 
Courant element approximation.  postpone until the Appendix the proof of an 
impol'tant coloring lemma used in 3.2 for tree approximation in the "'trim and cut'" 
algorithm. 
For convenience, we use the convention that positive constants are denoted bv 
c, c,.., throughout and thev may vary at everv occurrence. The notation A  B 
lneans that ClA  B  c2A. 



2588 B. KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY 

2. PRELIMINARIES 
In this section we collect ail the facts needed regarding nmltilevel triangulations. 
local approximation, quasi-interpolants, and other results whiçh were developed in 
[11] a.lld earlier papers. The essentials are presened for clarity but without proofs. 
2.1. iangulations. By definition 
çan be represented as h« ration of a finite set  of closed triangles with disjoint 
interiors: E = zeN, .  shall alwavs assmne thal lhere exists an initial 
triangnlation  of E of this fi)rm.  call 
a mullilev«l lriaguh#io¢ «»f E with levels (OE,) if lhe following conditions are ful- 
tilh,d: 
(a) l':v¢'ry lev¢,l OE,, is a lmrtiti«m 
AeT,,,  and OE,, c«msists ¢»f «h»sed lrialigles with disjoint interiors. 
(h) l'h¢' l¢,v«ls (ŒE,,) ¢»f ç are n¢'sed, i.c.. OEn+l is a refinement of 
(«) Ea«l li«mgle   , ]las al I'ast lwo and at most 310 children (sub- 
I i'img[(,s) in OE,z+l, wh¢,re .1I  4 is a constant. 
(¢1) l'he valence X,. of each vertex e of anv triangle   OE (the number 
¢,f the trianglcs ff'oto OE,, that share t' as a verex) is at most .OE, where 
  3 is a COllStallt. 
(e) No-ha¢gig-v«rli««s «omtitio: Ne» vertex of anv triangle   çm that 
I»elongs to lhe interior of E lies in the interior of an edge of another 
triangle from 
Xk' denote bv ç',,z the set of ail vertices of triangles from ,, where if v  ç is 
on the bCmndary of E, we inclnde in ç as many copies of v as is its multiplicity. 
With this mderstanding, we set ç= 
 now introduce three types of nnlltilevel nested triangulations which will play 
an essential role in our developments: 

• Locally regular triangulations. \Ve call a multilevel triangulation 2- = 
[.J,n_>0 OE, of E, a compact 1)olygolml (lomain in N 2, a locally regular triangulation. 
or briefly an LR-triangulation. if T satisfics the following additional conditions: 
(i) Thcrc exist constants 0 < r < p < 1 (r  ), such tllat for each L  ç and 
anv child ' of L that belongs to ç. 
(2.2) rl I  I'1  piPI. 
(ii) There exists a constant 0  3  i su«h that for each '. "  ç (m  0) 
with a. common vertex. 
(2.3)   la,,   . 
• Strong locally regular triangulations.  call a nlultilevel triangulation 
T = ,0 T of E, a compact polygonal domain in OE2, a strong locally regular 
triangulation, or simply an SLR-triangulation. if ç satisfies condition (2.2) and also 
the following condition (which replaces (2.3)): 



NONLINEA1R PIECEWISE POLYNOMIAL APPROXIMATION 2589 

(iii) AJfie transform angle condition: There exists a. constant /3 = /3(T) > 
0 (0 < /3 < -) such that if 0  OE (m, > 0) and A • R 2  R 2 is an ane 
3 -- 
transtrm mapping A0 one-to-one, onto an equilateral reference triangle, then for 
every triangle A ¢ T with a. conlnlon vertex with Au, we have 
(2.4) rein angle (A(&)) 2 fl 
where A(&) is the image of & under A and is therefore also a triangle. 

• Regular triangtalations. B.v &'finition a multil«-«el triangulati(m T of E C 2 
is called a regular triangulation if ç sa.tisfies the fi»llowing condition: 
(iv) There exists a constanl ç = ff(T) > 0 su«h that the minimal angle of each 
  T is greater than or equal to fl- 
The remainder of this subsection makes several observations to better mMerstand 
the natm'e of multilevel triangulations. First, it is clear that the classes of LR- and 
SLR-triangulations are each invm'iant ramer ane transforms. X> next observe 
that ea«h SLR.-tlimgulation is m LR-triangulation. but that the converse statement 
does hot hold. loreover. ea«h regular triangulation is an SI,R-triangulation, but 
again the converse is in general false. Çomtcrexalnples are given in [11]. 
Each type of triangulation dcpends on several parameters which are hot com- 
pletely independent. For instance, the paralneters of LR-triangulations are 3Io, , 
r, p, & and # (the caMinalitv of ). We could set al 1 
 = 7, P= 1-randelimi- 
nate these as parameters, but this would tend to obs«m'e the actual dependence of 
the estilnates upon given triangulations. 
We next brieflv dcscribe a siml)le standard procedm'e for conslructing lnultilevel 
triangulations. XX start ff'oto an initial triangulation  of the given compact 
polygonal domain E C N . We then select a point on each edge of every triangle 
   and join them within  by edges to subdivide  into four children. The 
collection of ail such children hecomes the first generation of triangles, which we 
denote bv . We recursively refine in this way to produce succeeding generations 
, ,.... The resulting collection T := U,0 OE, is a multilevel triangulation of 
E. 
It is important to know how the quantities ][, Inin angle(), and Inax edge() 
of a triangle   T mv change as  lnoves away froln a fixed triangle > within 
the saine level or through the nested refinements. Consider the case when ç is an 
LR-triangulation. Then conditions (i) and (ii) suggest a geometric rate of change 
of ]] (at the saine level). In fact, /ho rate is polynomial [11]. Fur/hermore, 
if '.A"  OE (m  1) have a COnllnOl vertex and are also children of some 
  T-I, then, as shoxw in [11], it is possible for ' to be equilateral (or close to 
such), but for " to have an uncontrollably sharp angle (see Figure 1). 

FIGU1RE 1. A skewed cell 



2590 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY 

If2- is an SLR-tria.ngulatioi, the above configuration is impossible, but the trian- 
gles froln 2- still lnay bave uncontrollably shaip angles. In this case, rein angle() 
changes gradually froln one triangle to the adjacent ones. 
For anv vertex v G çm (m  0), we denote by 0 the cell at level m associated 
with v. i.e., 0.,. is the UlliOll of ail triangles from OE that have v as a common vertex. 
We denote by , the set of ail such cells 0 with v G ç, and set  = Um0 oto. 
2.2. Local piecewise linear approximation and qui-interpolants.  d 
note by Hk the set of ail algebraic polynomials of total degree less than k.  shall 
often refer to the following lelmna (see [11]), which estat)lishes the equivMence of 
differcnt norms of polynomials over different sers. 
Lemnla 2.1. Let P  ll. k  1, and 0 < p,q  . 
(a) For ang tr'iaogle  C 2, 
(b) If A aod A' are tu,o trian91cs such tbat A' C 5 and I1 
(c) tf a' c 5 .., la'l  qll ,,,itn o < «, < 1, tben 
) 1 1 
Io tbe above e.rpressions, tbe coo.stants depeod at mo.st on tbe coespondm9 param- 
tiers and tbe constant c. 
l'he no-hanging-verlices condition (e) of triangulations guarantees the existence 
»f ('ourant elements. Namely, for any vertex v 6 ç (m  0) there exists a 
unique Courant element ç0, supported on 0 G , which is the unique continuous 
piecewise linear fllllCtion Oll E that is supported on 0, and satisfies ç0,. (v) = 1. We 
denote 6 :=  := (0)0¢0- We also denote by S the space of ail continuous 
piecewise linear flmctions over T. Clearly, S E S if and only ifs =  S(v)ço.. 
Throughout the remainder of this section, we assume that T is an LR-triangulation 
of E. X shall often use the tbllowing stability estimates for 
Lennna 2.2. Let 0 < q   attd S =  ao ço, m  O. witb coecients ao 
Then for every  G Tre, we bave 

and hence 

IISIl,  (  
OEO,, ODA 

I 
0 
wzth constants of equivalence dependin9 only on the parameters of 2-. 
estimates the q-norm is replaced bg the sup-norm if q -- oe. 

In tbese 

The proof of this lemlna is fairly silnple and ca.n be found iii [11]. 

Local piecewise linear approximation. The local approximation by continu- 
ous piecewise linear functions will be an ilnportant tool in out flrther development. 



NONLINEAR PIECEWISE POLYNOMIAL APP1ROXIMATION 2591 

For f E L(E), q > 0, and any A E "Ym (m _> 0), we denote the error of 
approximation to f from S by 
(2.5) (f) := (f, T), := inf I[/- SIIL(a), 
SSm 
where  is the UlfiOn of all triangles rioto OE, that have a vert.ex in common with 
• Quasi-interpolants. The set y of ail Courant elements is obviouslv redundant. 
To obtain a good (i.e., sparse) representation of a given fimction f, we shall use 
the following well-known quasi-interpolant: 
(2.6) Q,(f) := Q,,(f, 7) :=  (f, 
0 
where (f, 9) := lE f9 and (ç0) are the duals of (ç0) defined bv 
(2.7) « :=   
AEOE.,ACO 
9 the "central vertex'" 
with ,e the linear polynomial that is cqual to  at 
0, and equal to ,A at/he o/ber two vertices of  (recall that N, is the valence 
of v). It is easily seen that 
(çe, e') = 500,, for 0.0' E . 
Obviously, Q is a linear projector, i.e., Q(S) = S for S E Sm. It is crucial 
that e E L and e is locally supported. Conquently, Q is locally bounded 
and provides good local approximation. 
Lemma 2.3. (a) If f  Lv(E ). 1 5 q 5 , ad  E ç, m OE O. then 
IIQ,(/)II.()  cll/ll.(n)- 
(b) IfO < ç   and g = e  " P witb P E H2 and m  O. tben 
IIQ.(g)llL.()  clgll.(n), for 
The constats above depend oly on ] and the parameters of 
For a proof of this lenmm, see [11]. 

From the above lemlna, we see that Q,, : L,(E) -- S,, (1 _< l <- oe) is a locally 
bounded linear projector. There is a well-known scheme for extending Q to a 
nonlinear projector Q : Lo(E)  S, for 0 < ri < 1. This is needed for nonlinear 
approximation in Lp (0 < p  1). To describe this extension, let PA.ç : Lo(A)  H 
(0 <   ) be a projector (linear if ri  1 and nonlinear if 0 < q < 1) such that 
IIf- P,(f)ll.()  cE(f,A) for f E L(A), 
where E2(f, A) is the error of the best L.(A)-approxilnation to f froln H2 (the 

linear polynonlials). We define 
.(f) := 

and set 
(2.8) 

Z 11A. 

T,,o(f) := Qm(p,,o(f)), for f  Lo(E ). 



2592 
Clearly, ,., • 
0< r/< 1). 
The lmXt lelnlna, established in [11], shows that Qm and Tm,., provide good local 
approximalions Dom 8. 
Lemma2.4. (a) ff f  L,(E). 1 5 1 x, ad A  ç, m  O. the 
(b) lf f  Lo(E ), 0 < 1 ,, artd A  T, m  0. then 
lift- T,(fl)ll()  c(fl)o. 
The coustats above doperai ozdy on 1 aztd the parametcrs of T. 
The needed convergence of Q,,,(f) and T(f) to f is provided bv tlle following 
res, dt (sec l,emma 2.15 ff'on, [11]). 
Lemma 2.5. {f f  L,(E). th«t 
Now. we apply a well-known schtullo fi,r ohtaining sparse Courant elelnent re 
resentation of flm('tions. b defiue 
(2.9) qm := Q, - Q,n-t aud tre.-0 := T.o - T-L-o. for m R 0. 
where Q_ := 0 and T-l.o := 0. Clearly, qm(f), tm,o(f)  m. 
For a given flmction f G L0(E ), 1  q  OE. we define the sequence b(f) := 
(bo(f))oeo= flore the expression 

B. KAIRAIVANOV, P. PETIRUSHEV, AND IR. C. SHAIRPLEY 
L.,(E) -, 8m is a projector (linear if 1 >-- 1 and nonlinear if 

(2.10) q.m(f) =:  bo(f)9o, m >_ O. 

Using Lennna 2.5. we have 

(2.11) f =  q,(f)=   bo(f)c2o in L,. 
m>_O m>_O OEO, 

If f C L,(E). 0 < '1 < 1. we define the sequence b,(f) := (bo.o(f))oeçm bv 

(2-1 ')) tm,o(f) =:  bo.-o(f)o, m >_ O. 
0E(9,. 

and again by Lemma 2.5, we have 

(2.13) 

m_>0 m_>0 

Clearly. b(-) is a linear operator while b(-) (0 < r/< 1) is nonlinear. 



NONLINEAtl PIECEWISE POLYNOMIAL APPROXIMATION 2593 

2.3. B-spaces. In this section, we include the necessary tools for the B-spaces 
which we need for nonlinear -t, erln ('ourant element aI)proxination. The B-spaces 
over multilevel nested triangulations of IR 2 are introduced in [11] and used there for 
nonlinear n-term Courant element approximation in L»(IR 2) (0 < p < oc). Iu the 
present paper, we shall use the B-spaces for n-terre Courant element approximation 
in Lp(E) (0 < p <_ o), where E is a compact polygonal domain in IR 2. We shall 
put the emphasis on approximation iu the lllliforlll llOrlll (p = Oo). There are three 
t.vpes of B-spaces (skim.v, slim, and fat B-spaces) that w('re introduced in [11] to 
serve different pnrposes. For ('om'ant (,lement apl»roximation, we need the slim 
B-spaces, which we shall simply call B-spaces. 
Throttghout this paper, we assume that "Y is an L/?-triangulation of a compact 
polygonal domain E in IR . Moreover, the B-spaces B(T), with parameter set 
1/r := a + 1/p a.ccording to two specific choices: (a) p = oc and a _> 1: or (b) 
0 < p < .;x and a > 0, will arise naturallv in out algorithms and error estimates. 
These spaces have several equivalcnt definitions, which we briefly descrihe. 

•Definition of B(T) via local approximation. We (lefine B(T) as the set 

) l/r 

where SA (f) is the error of L(f=x)-approxilnation (local) to f ff'oto $,,, for A 
(see (2.5)). It is readily seen tha.t If + g  f5 + gSç with < := i,{r, 
and If + s]B; = IfIB for s e &. Hence l" ]Bi is a Selni-norm if r  1 and a 
semi-quasi-norm if r < 1. 
By ThcoreIns 2.7 and 2.9 bclow, it follows that if f  B(ç), then f  Lp(E). 
Therefore. il is naturat to dcfine a (quasi-)norm in B (ç) by 
More generally, for (I < /< P, we &,fine 
(2.1Q Na,(/, ) := II/ll + (lll/v-/a(f))  
Evidently, N«(f, T) = IIflIB(z). Vhel, clear from the context, we use 
. Definition of norm in B(ŒE) via atomic deconlposition. For f  L(E), 
we define 
( 
(2.17) N(f) := inf (IOl-llcoollV 
Y=0eo coco 
where the iIfiInUm is takeu over all represent.alions f = 0eo c00 in L(E). Note 
t.hat the existeuce of such representations of f follows by (2.11) and (2.13). By 

Theorem 2.7 below, 

(IOl-llcooll)  <  implies  Icoo(')l  < , 
0 

and hence f c= Lp(E) and the series oo Ic00(.)l converges a.e. and in Lp(E). 
Therefore, the way in which the terres of the series are ordered is hot essential. 



2594 B. KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY 

(2.18) 
If p = oc, then 

and the convergence in L-(E) implies a stronger (absolute) convergence in Lp(E) 
(- < p). Bx Lemma 2.1, it follows that 
N(f)  inf ((o]l/p]co])) 1/r- 
f:0eo coco 00 
k / 
f=eo c0 

N,» (f)  inf Icol"- 
f=-oeo cotpo 
• Definition of norms in B(T) via projectors. For f  Lo(E ). we let 
(2.19) f =  bo.,(f)ço 
I,e lhc repreSelllatiol ,,f f ri'oto (2.11) if q  1 and from (2.13) if 0 < ç < 1. 
defiue 
(2.2o) x.(f) := (o-llbo«(f)çol)  
0 
and. more generally (in aceordanee with (2.16)). 
(2.21) NO,o(A) := (lOl/P-/°llbo.o(f)çollo)  
By Lemmas 2.1 aud 2.2, we have 
(2.22) No.o(f)  (IAl/-/°llq(f)llL.)y if ç  1, 
(2.23) NQ.o(f)  (IAIVP/°llt.o(f)llL,,¢a)) r if 0 < q < 1. 

and 

(2.24) NQ,,(f)(oe(lOI1/p[bo,,(f)l)r ) 
In the most interesting case of p = 

(2.25) 

• General B-spaces. 

A more general B-space Bq(T). c > O. 0 < p. q < oc.. is 

defined as the set of all f E Lp(E) su«h that 
f=-oeo coco OeO. 2- 1o1<2 -m+l 



NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2595 

where the fq-nOrln is replaced by the sup-nonn if q = oe. In this paper, we do hot 
need the B-spaces in such generality. 
• Embedding theorems and equivalence of norms. e recall our assump- 
tions.  have 0 < p 5 oe, and   1 if p = oe and a > 0 if p < oe. In both cases, 
1/w := + 1/p (1/w :=  if p = oe). We record estilnat.es and embeddings froln [11], 
along with the necessary lno(lifications, which me necessarv for the development of 
the main results of this I)at)('r. Tho first oml)ed(ling rosult al)t)ears as Theorem 2.16 
in [111. 
Theorem 2.6. For O < r < p or p = oe, r  1. tben for ar sequence of real 
unbers (co)oe. we bave 
(2.26)  I«lç  « I1«11; , 
OGO P - 
whe c depends onlg o . p. and tbe parmneters of 
Theorem 2.7. lf f  Lu(E ) with O < l < p. and NQ.(f) < oe, then f  Lv(E) 
(f  C(E) if p = oe), ad f bas tbe repsetation f 
series com,er9in.q absolut,'ly a.e. i E and in Lp (respeetivelg. in C(E)). 
(2.27) IIfllp  ll lbo.n(f), oll 
wheve c is idependent of f . 
Pvoof. For 0 < p < oe, the result follows from (2.11). (2.13), and Theoreln 2.9 
below. If p = oe, the theorem follows by (2.11), (2.13). (2.25), and the following 
estimates: 
bo,v(f),ço  Ibo,v(f)l  
Remark 2.8. It is easilv seen that Theorem 2.7 is hot true when p =  and a < 1. 
For this re&son we impose the restriction a  1 when p =  throughout. 
Theorem 2.9. The noms ][. ]]B7(), Na,v(-) (0 < V < P), N(.), a,,d NQ.v(.) 
(0 < ç < p), defined in (2.15). (2.16), (2.17). and (2.21). are equivale, t with 
constants of equivalence depending only on p. 
Proof. One proceeds exactly as in [11] (see the proof of Theorem 2.17 of that 
reference) and proves that 
l/l>  (  (lll/v-/'aa(f),)) / 
f=o¢e coo O o 
provided 0 < W < p. To obtain the norm estimate from these semi-norn equiv- 
alences, we use Theorem 2.6 to give I111  c(). çsing this, (2.28), a,d the 
renmrk after the definition of N(f) iii (2.17), we obtain 



2596 B KAPAIVANOV, P PETIRUSHEV, AND 1R. C. SHA1RPLEY 

For the reverse inequa.lity, we use Lemlna 2.2. Theorem 2.6, and (2.28) to obtain 
/- ) /P 
(o "bo,(f)o";)  c(o, ,p) (o "bo.(f)o" 
c < 
cl[fHp+«(  
 cfe(z. 
This and (2.28) iml,ly ,,(f)  c[Iflla(z). 
l'he next embedding tlworem of Sobolev type foliotes immediately from (2.18) 
«»1" (2.24). 
Theoreln 2.10. kbr 0 < no < « and ri := 
«ontinmms cmb«ddin 9 
"" T) 
(2.'2) B7)(7) c B ( , 

• Interpolation. \Ve first recall some basic definitions from the real interpolation 
method. ], refcr the rea,l«r to [2] and [1] as gencral references for interpolation 
thcorv. For a pair of quasi-normed spaces Xo, Xt. emhedded in a Hausdorff space, 
the space X0 + X is defined as the collection of all flmctions f that can be repr 
sented as f0 + f with f0  X0 and f  X. The quasi-norln in X0 + X is defined 
IIfl[x+x, := [If[xo+xl + i,,f ]lf011x + I[fxll.x,. 
f:fo+fl 
The K-functional is defined for each f  Xo + X and t > 0 by 

(2.30) /x-(f,t) := Iç(f,t;Xo.X):= inf IlY011x0 +tllflll.xl- 
f:fo-}- fl 
The ïeal interpolation space (Xo, X), with 0 < A < 1 and 0 < q  x is defined 
as the set of ail f  Xo + X sueh that 
(1 )l 
where the Lq-norm is replaced bv the sup-norm if q 
It is easily seen that if X C X0 (X contimously embedded in Xo), then 
K(y.t)  IlYllx for y  x0 a.d t  1, .d. consequently, 
(2.31) ]f(Xo,X,),  ]IYlIx + [2"Xlç(f.2-")] q 
Theorem 2.11. S'appose 0 < p   and fuether assume that bolh o,   1 in 
the case p = .. and o,  > 0 otherwise. Furthermore. let  := (oj + l/p) -. 
j = O. 1. 
(2.32) 



NONLINEAR PIECEVvlSE POLYNOM1AL APPROXIMATION 2597 

with «quival«rd hOtms, provid«d a = (1- A)ao + Aa with 0 < A < 1 and - := 
(et + l/p)-'. 

Proof. We shall prove (2.32) only in the case p > 1. For a proof of (2.32) when 
p _< 1, see [3]. 
We shall use the abbreviated notation B  := B(7-) and B% := B- (7-), j = 
(}, 1. Also, we denote by gq the space of all sequences a = (a0)0e of real numbers 
such that Ila'l«. := (E I1 ) '/" < - 
We set / := 1 and non,,alizc the C(mrm,t elemenls iii Lp, tl,at is, II0]lp -- 1. 
we also renorlnalize the duals qS0 rioto (2.7) accordingly. \Ve denote again by 
b(f) = (bo)oee) the sequence ri'oto (2.10) witl, respect to the ,on,,alized Courant 
elelnents. By (2.24), Theorem 2.7, and Theorem 2.9, if f  B , j = 0.1. then 

(2.33) 

f = Z bo(f)2o and 

recalling that thc clemcnts ç)0 arc normalizcd ill Lp. Thc corresponding statemcnt 
holds for functiols f G B a as wcll. 
b shall next employ the following intcli)olatiol theorcm (sec. e.g., 5.1 of [1] or 
[2]) which follows dircctly froln the dcfinition of thc N-fimctional and the llOrlllS of 
the interpolation spaces. Suppose T is a lincar operator which bomdedly maps X0 
into tb and X into }, where (X0,-¥) and (lb, }) are couples of quasi-normed 
spaces as above. Then for 0 < A < 1 and 0 < q  , T boundcdly lllaps (-¥o, X1),q 
illt.o (}b, }),q- 
Xb introduce lincar Opcl'ators Z and P a.s follows: Z is dcfincd by Z(f)o := bo(f), 
0 G . and P is givcn l)y P(a) := oeoaoço, a = (ao)oeo. By (2.33), IIb(f)ll«.,  
cllfllB for f e , j = 0.1. and hcnce Z • B"  g (boundcdly). Bv the 
above-lnenIiolmd interpolat ion t heol'eln. 

(2.34) 

Z. (B °,B)«  (go,()« (boundedly). 

Similarly, if a G (,-, then bv Theorelns 2.7 and 2.9. we mav conclude that 7)(a) L,,= 
Y0eO a0ç0 is well defined. So if we set f = 7)(a), then 

[[P(a)['B <_ c inf [[(co)o,,e. <_ c,,a[[t.. 3 = 0.1. 
f=}2oee cooo 

Thus 7 ) " g'5 --' B (boundedly). and by interpolati(m 

(2.35) 

7)" (gro" ç,)),r - ( B'°. B*))« (boundedly). 

Finally, we recall the well-known interpolation result (see, e.g.. [2], [1])" 

(2.36) 

(Otro,(rl),Mr = tr, where 1 _-- 1-k _1_  with 0 < A < 1. 
T T 0 T 1 

Clearly, (2.32) follows by (2.33)-(2.36). 

• Skinny B-spaces. The skimy B-spaces were introduced in [11] and used for 
characterization of nolflinear (discontinuous) piecewise polynolnial approxilnation 
on IR 2. \Ve next adapt that defilfition to the case of approximation on a compact 
polygonal dolnain E C ]R 2. Supposc 7- is a multilevel nested triangulation of E 
whicl, additionally satisfies condition, (2.2) (sec §2.1 and [11]). The skimlv B-space 



2598 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY 

Bk(T), vhere k _> 1 and c and "r are as above, is defined as the set of ail f  Lr(E) 
such that 
) l/r 
(.aT) s(, :: (ç(-(s.))" < , 
where w(f, ) is a kth modulus of smoothness of f in L(), defined by 
wk(f,)r := sui, II(f.)ll.() 
and (f,-) is the kth difference of f. The norm in (T) is defined by 
* Fat B-spaces: The iink to Besov spaces. Sul)l)OSe T is an SLR-triangulation 
of a compact l)olygonal donmin E C 2. Similarly as in [11], we define the fat B- 
spacc (T), whcre k  1 an(1 a an(l r are as al)ove, as the set of all fimctions 
J  L(E) such tha.t 
(-asl II(z := (ll-(I.a))" < oe- 
thcorcm, it rca(lilv fi,ll(,ws that Cl2(f, L)  &(f)  c2w2(f, &)r, and hence 
]f8/=(7)  clfIBv(T)  clf]=(7). The st)ace 2(T) is a natural candidate to 
rcplace B (T) in nonlilmar -tcrm Courant element approximation. This is, how- 
evcr. only possible for suciently snlal] a (0 < a < a0). Otherwise 2(T) is too 
"'fat" and cannot do the job. Finally, we note that if T is a regular triangulation 
and 0 < a < k, thcn a(T) coincides with thc Besov space B(L). For a more 
complete discussion of this and other related issues, see [11]. 

3. ALGORITHMS FOR n-TERM COURANT APPROXIMATION 

• Decomposition step for ail approximation algorithlns. The first step 
of each of the three at)proxilnation algorithms that we consider in this section 
is a decolnl)osition step. This stcp is hot trivial, since the set (I'7- := (0)0eo 
of ail Courant elements is redundant and. therefore, each function has infinitely 
many representations using Courant elements. For each algorithm, it is crucial to 
have a sufficientlv efficient initial representation of the fimction f that is being 
approximated. This means that the representation of f should allow a realization 
of the corresponding B-norm. 
To construct the initial representation, we consider two cases of metric approxi- 
mation. If the approximation takes place in L u, 1 < p <_ :x, we utilize the decoln- 
position of f via quasi-interpolation from (2.11) with 1 <_ q < p, while if 0 < p < 1, 
we use (2.13) with 0 < 1 < p. In both cases, we bave an initial desirable sparse 
representation of f of the form 
(3.1) f = Eb°ç°' bo = bo(f), 
which allows a realization of the B-norm (see (2.24) (2.25), and Theorem 2.9). For 
the remainder of this sectiou, iii order to more easily track the dependency of the 



NONLINEAIR PIECEWlSE POLYNOMIAL APPROXIMATION 2599 

constants appearing in the inequalities, we redefine Il fllB(ï) by 
(3.2) II/IIBa() 
which is an equivalent norm in B (T) (see Theoreln 2.9). Without loss of generality, 
we may aSsulne (when needed) that there is a final level (-)L (L < oe) in (3.1). 
3.1. "Threshold" algorithln (p < oe only). In this algorithln we utilize the 
usual thresholding strategy used for o-erm allroxilllation ri'oin a basis in Lp (1 
p < oe). The resulting procedure performs extremely well. due to the sparse re 
resentation realized bv the first step. k" note, llowever, that the derived error 
estimates involve COllstants that depend on p and becolne Ullbounded as p 
The "push the error'" and "trim and cut" algorithlns desçribed later in this section 
will be shown to achieve the correspollding estilnates for the Ulfifornl IIH'III (p = ). 
For this subsection we therefore assulne that f 

• Description of the "threshold" algorithm. 
Step 1. (Decompose) Wc ue the dec(,nll)osition of f E Lp(E) froln (3.1). 
Step 2. (Select the  largest terres) b order the terres (boo)o6o in a sequence 
(bo2 ço) so that 
(3.3) Ilbolço, ll  IIboçoll  ... 
Then we define the approximant T T n 
.4. (y)v y a= eçe. 
An (f), := 
• Error estimation for the "threshold'" algorithln.  denote the corre- 
sponding error of approxinmtiol of this threshold a.lgorithnx bv 
T T 
The argument used in establishing the Jackson error estimate in [11] may be mod- 
ified in obvious wvs to prove the following error estilnate. 
Theorem 3.1. IffG B(T). >0. 1/ç::+l/p (O<p<oe), then 
(3.a) T 
where c depends on . p. and the parameters of T. 
In 5, we shall need the following result: 
Lemma 3.2. If f : o boço is the decomposition off from (3.1). then 
j- 1 
where (boço)= is as in Step 2 and c depends on . p. ad the parameters of T. 
Proof. Applying Theorem 3.4 from [11] to (bo9o)=,+ immediatelv provides the 
desired result. 

Remark 3.3. As we have mentioned, the main drawback of the "threshold'" algo- 
rithln is that it is hot applicable to approxilnation in the unifornl norm, since the 
constant c = c(c, p) in (3.4) tends to infilfity as p - oe and the performance of the 
algorithln deteriorates as p gets large. The obvious reason for this behavior is that 



2600 B. IxAIRAIVANOV. P. PETRUSHEV, AND IR. C. SHARPLEY 

f can be built out of many terres (boço) which have small coefficieuts and are sup- 
ported at the smne location. These terres can pile up to an essential coutribution. 
but the algorithm will rail to antieipate their fiture significance. 
3.2. "im and eut (the tree)" algorithm. The idea of this algorithIn h its 
origins in the proof of the Jackson estimate in [7] (see 5. pages 272-276). The 
approxima.tion considered there is hv wavelets or splines over a uniform partition 
in the unifornl llOrlil.  shall refillC this i(lca to devclop ail algorithm for n-terre 
Colalt cleinent apl»I'OXilnatiol in Lv(E), 1 < p  , over L-triangulations.  
begin with a ln'ief desci'iptiol of lhe algorithm and then elaborate on the details of 
«a«h of the inain steps. 
• Description of the "triln and eut" algorithm. 
Step 1. (De«ompose). ki, us« thé, «olmnon decompositiou of f  Lp(E) given 
ill (3.1). 
Step 2. ( ()ylonize th« colis q[" O into man9eable trees ").  develop an al- 
gorilhm (pro«e«Iure) for ««»l»ring Ihe cells »f (-) in such a wav that the 
colis of the smne colon" fi»rm a Ir«e structure as deseribed in Lelmna 3.4 
I»«l«»w. This organizaIim greatly simplifies Ihe lnanagement of the es- 
limaIes, b«»lh Ihe aI»proximalion coustru«tion and the enumeration of 
"active" Courant elelnents in our ai)proximant. 
Step 3. (Trim «ach tr«e). Sin«e all the elements lnay iuitiallv affect the B- 
spaee n«»rm of a flmction, we need to preprocess each tree by pruning 
alI t»ranehes whieh mav havc manv leas, but do hot make a significant 
contribution to thc norm of the flmctiou f, X do this bv running a 
stopping tilne argulnent from the finest level to a coarser level, until a 
significant cmnulative comribution is met.  prune the branch just 
below that element. 
Step 4. (Partition the remainin9 trees into "'segments"). We continue to par- 
tition the remainders of each of the Iç trees bv cutting them at each 
of the joins of branches to form chains rioin the tree. We will easilv 
be al»le to track the numi)er of chains produced bv this procedure. A 
second stoI»ping time argument is then applied to eut the chains iuto 
• "segments'" in ord«r to coutrol the uumber of significant eleluents added 
to the approximant (at most N0 + 1 from each segmeut) and to guaran- 
tee that the culnulative effeçt of the left-over elements (i.e., error) tan 
be controlled bv the final Step 5. 
Step 5. (Rewrite the "'se9ments'" fo control error). Here each segment is rewrit- 
ten at its fiuest level, and its terlninal element (with the new coefficients) 
and some of its neighboring elements are added fo the approximant. 
This allows for a void to be created, so that the residual of the segment 
will have disjoint support with all remaining segments  well as the 
residuals of those previously processed. This insures that. the cumula- 
tive pointwise error remains under control. 

We now describe these rather vague steps in more detail. 
our earlier discussion. 

Step 1 is clear from 

Step 2. In the followiug lemnla, we construct a procedure for coloring the elelnents 
of (3 with K colors u, so that no two Courant elements of the saine color from the 



NONLINEAR PIECEW'ISE POLYNOMIAL APPROXIMATION 2601 

saine level have SUl)l)orts that intersect; in fact, cor1espondillg cclls of the sanie 
color will bave a tree structure with set inclusion as the order relation. This allows 
us to partition  into a disjoint milan of sets " (1  u  Iç), and correspondingly 
K 
organize f as the sure f = =lf, where f := 0eo" boço. b tan then 
proceed to process each of the f, without vorrying about its terres from the saine 
level overlapping, and at worst a factor of K will came into the constants for the 
estimates that we dcrive. For its proof, sec tllc Appendix. 
Lemlna 3.4 (Coloring lcnuna). For ay multilevel-triaogulation T of E, lhe set 
 := (T) of all cells geterated bg T con be represc,lcd as a .fitite disjoint io, of its svbsets (O"))1 with Iç = Iç(N0, M0) (No is te maximal valence ad 310 is 
the maximal umber of childrn of a triangle la T) such that each 0'" bas a lree 
structure with respect to set inclusion, i.e., if 0 . O"   wzth (0) °  (0") °  O. 
the either 0  C O" or O" C 0 . 
In order to COml)lcte the remaining Stcps 3-5 we must consider two varialions 
iii the details of the algorithln, (lepell(ling on whether p =  or 0 < 1 » <  Thc 
case of thc mfiform mctric is plcscnt(,d in Subsecti(m 3.2.1, while thc case of Lp 
(0 < p < OE,) is giron in Subscction 3.2.2. 
3.2.1. The ca.se p = ,. Fix e > 0 and let e* .-'- , « whcre we recall that Iç is the 
nmllber of colors rcpl'esenting the tlee structures. 
Step 3. Trimming of 0'" (1  oe  Iç) with e*.  triln each ", starting ff'oin 
the finest level  and procecding to the coarsest level.  remove ff'oto " everv 
cell 0 ° snch that 
(3.5)  Ibol  *. 
0C0 ° 
 denote bv F" the set of ail 0  " that bave been retained after completing 
this procedure, and by Fï the set of all final cells in F , i.e., 0   Fï iff there is no 
0  F  such that 0 Ç 0 . Clearly, for each 0   Fï, 
(a.6)  Ibol   for each 0'  0 . but  Ibol > *. 
0C0  0C0  
 denote fr := 0er boço. Therefore 
(3.) II/ -/11  max II  booll  mx  Ibol  *, 
0°F 0C0 ' 0°F" 0C0"- 
K 
and hence, if we set ff := = fr, then 
(a.8) II/-/11  tç* = /oe- 

Step 4. Partitioning lhe branches of each tree F v into chains aod lhe chains into 
"'segments". For each of the tree structures F" (1 <_ , _< Iç). we denote by F the 
set of all branching cells in F" (cells with 1note than o11e child in F') and by Fo" h 
the set of all chain cells in F" (cells with exactly ana child in F'). It is easv to see 
that 
(3.) #r < 
In fact, o11e proceeds bv induction ff'oin the filleSt o coarser levels, associatillg each 
branch cell ff-oin F by a cell froln F}. For each branch cell, there is alwavs at least 



2602 B. KARAIVANOV, P. PETF{USHEV, AND R. C. SHAF{PLEY 

one meniber of F} still available flore each descendant edge. Only one is used to 
associa.te with the current i)ranch cell, thereby leaying at least one available for its 
Imxt ancestor bran«h cell in that line. 
On the other hand, #Fh may be nmch lm'ger than #Fï, and so we will need 
to process these elenmnts. A collection of cells 01 D 02 D --- D 01 is called a chain 
if ibr j : 1 ..... ! - 1. Oj+ is a child of Oj and 0  Fn, and the terminal cell 
0t  Fï  P.  partition the tree F  into chains. Namely, we start at the coarsest 
level and construçt (maximal) chains whiçh will ternfinate with either a final cell 
(in l'ï) or a branching cell (in F). XX contime this procedure to the finest level. 
Xb nex[ "section" each «hain into segments using e* as a threshold. Namely, if A 
is a chain and A : (03 )=1 with 01 D 02 D "-" D Or, then we start frOlll the coarsest 
eh'ment 01 mM smn the «oecients of each cell, moving to the next child of the 
«hain mtil the sure exceeds the threshold. At this point we cut the chain to form 
the first (significalt) segment and start this procedure again with the next child in 
line mitil this is hot possible (i.e.. ending without the threshold being crossed). X 
«ail this type of segment a "r«mn«nt segment". Therefine, this procedure cuts A 
into disioint segnwnts a »f thc finm (OjV +u 
• ,j=i, I t  O. so that each segment satisfies 
exactlv one of the following conditions: 
(a) a «onsists of a siugl« «significant et,Il"" 

(3.IO) 
(6) 

13.11) 

(3.12) 

[bo, ] > e* (case ,,f p = 0), 

 IV, I > * (c.e of # > 0), 
j=i 

I 
, Ibo, ' < 

i+p 
\Ve denote bv Z ' the set of all such seglnents o- = (Oj)j= i 
procedure. 

resulting [rOlll this 

Step 5. Reu,riting elcments from certain segments of '. Let a = (Oj)j= 1 be alw 
seginent ffOln Z, and suppose that the finest cell Op of 
rewrite the ('OUlalt elelnents 
()=l boço) of the segment at its finest (m-th) 
hwel. finding coecient.s (co) such that 
 COçO : bojço on 0. 
OOm,O°O,#O 1=1 
XXdenote.« := {0  O : 0 0 # 0and0 C 0}. Obviously. ifa = 1 (i.e., 
the segment consists of a single cell), then the coecient relnains ulmhanged and 
 = a = {01 }. Observe in any case that #  N0 + 1 and Uoex o c o. Finallv. 
set  := =1 , and correspondingly define 
(3.13) AZc(f) := 
aEN OEA 
as our approximant produced by the "trim and eut" algorithln. 



NONLINEAR PIECEWISE POLYNOMIAL APPFIOXIMATION 2603 

• Error estinlation for the "trilll and cut" algorithm (case p = oc). 
Suppose that the "triln and cut" procedure has been applied to a fmlction f with 
4TC( ) 
 > 0, and . e -f = 0A c090 is the resulting al)proximalt from (3.13), wheIe 

A« = U«ez ";» We denote 
,,(e) := ,,±(«):= #A, 
and 

TC 
A.()(f) := IIf - AyC(f)[[, 

ln TC(f)oc := illf { TU } 
A()(f) "n(e) _< n . 

Note that each of these quantities depend ilnplicitly Oll 7. To COlnl)lete Ollr results 
for the "'tliln and cllt'" algorithln, we show fiI'St in Lelllllla. 3.5 that this is a good 
approxiInation to f, and then that the lmnfl)er of elelnents that are used in the 
approxillmnt satisfies the correct estinmtes (see Theol'eln 3.7 below). 

Lennna 3.5. Suppoe tlat ATC(f) is tle appro.rirnant for f given in equalion 
(3.13) whi«h bas been constr«'ted using the "'trm ood c,l'" algorithm. Then 

(3.14) 

Proof. Followil,g the definitiol, (3.13) of AyC(f), we (l('fille 

A":=  coço. 
c,'E" 0 A', 

Then obviouslv. AC(f) = Eu=l/ A". SillCP g* - 2-,e it suces to shoxv that 
In Step 5 we extracted the heart of each segment a = (Oj)j=, added its con- 
tribution to tlle approxilnant (3.13), and cleared rooln for descendant cells. To 
estilnate the sociated error, we introduce tlle ring for a   := 0  0,; then 
R« = 0 when a consists of a significant cell (i.e., condition (3.10) hokls). For any 
llOlmmpty ring « (  "), set ' := (0) and observe that at worst 
L(01) 
It is easv to see that all rings R« (a  ") are disjoint and the set where A" mav 
differ froln fr is contailmd in «e£ 
and then over all colors u, it follows that 
K 
lift rc 
-4 (f)l,    llf- 
This together with estimale (3.8) ilnplies the desired error estimale (a.l). 

Remark 3.6. Conditions (3.5), (3.11), and (3.12) can be relaxed by replacing every 
smn  Ibo[ bv I[  boço Iloe. This would hot change the rate of approxilnation but 
lllay ilnprove the constants in a practical ilnplenmntation. 



2604 B. KAFIAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY 

Theorem 3.7. lffEB(T),a_> 1. ç:=l/a, lhenforeachs>O, 
(3.16) TC 
A.(«)(f)oe _ ¢ and n(¢) 
whcre c = c(Sç, ,lv, «). Therefore, 
Pro@  have ah'cadv shown in Lelmna 3.5 that NTC  ¢ < ¢: so we onlv need 
to estal,lish («) 5 c e-'llflla().  first obs,rve that it is cnough to estimate 
VEr, since «ontributions to thc approximmt oc«ur onlv as ea«h segment from 
is processed. Note that at most olp clement is «ontributcd for segments consisting 
»fa singlc signifi«ant eell (3.10) and at most o + 1 «ontributions for the segmets 
satisfying instead cither (3.11) «)r (3.12). 
I ordcr to estilnatc #Eu we first estilnate #F Ï, sin«e it will estilnate certain 
Icrms. Thc stq)ling criterimn (3.6) in Stcp 3. 
t3.1s) * <  Ibol. 
0C0 o 
must hold t«»r each 0'»  I" Ï. No if wc al»lfly the ç-th power to both sides, use the 
cml)(,lding of thc scqu«,nc, spa«es (ç  1), smn over all 0  Fï, and observe that 
Ihe snl)l»orls of th« ('«lls in I') havc di}oint interiors, thon we obtain 
(3.)) #I'ï (e*)  <  
l'hc rightmost incquality follows immediatclv bv our dcfinition of the nornl of 
B;'(ç) (ce (a.2)). 
To conlplete the proof of the theorem, we OlllV nced to cstablish a similar estimate 
tÇr Ihe lmmbcr of elcmcnts of E u. Rc«all, hoV(,l', that the segments « are %rmed 
as disjoint segmcnts of cells ff'oto thc tree structure and eome as one of two types. 
Esig. thosc cx('ceding the thrcshold (sec conditions (3.10) or (3.11)) and. Erem. those 
that do hot (sec condition (3.12)). KlOm thc construction it follows that renmant 
scgmcnts terminate with (qthcr a unique final «cll or a torique bran«hing oeil. and 
so bv (3.9), 
(a.0) #Zrcm  #r + #Fï  'e #Fï, 
which has just l)een shown in (3.19) to satisfv the dcsired bomd. 
Thel'C%rc wc arc reduced to estimating 
timating #F Ï (sec (3.18)-(3.19)) mav be elnploycd once again. Indeed. we just 
replace the condition (3.18} with 
(3.21) e* < lbol, 
and use the fact that the segments are disjoint (considcred as part of the tree 
structure), in order to obtain 
(3.22) g (*) <   lb01 r 
Although hot required hele, the following lemma will be needed in 5 and tan 
nov be established using the techniques of this section. 



NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2605 

Lemnla 3.8. Let f = ft,+ fl, where f = -oE6)boço, .fJ = -oo bô(;o (3 = O. 1) 
 := Ibl ) <  U = o. ) 
with oi  1 and ri = 1/cj. If the "'trim and ct'" algorithm with e = eo + e 
(zj > O) bas b«cnappli«d to f . r«pwsent«d as obove in place qf Stop 1. then 
(3.23) Te 
.¢+,)(f)  eo + 

and consequentlg 
(3.25) c(f)  «,,-o + c,,-,,ç], ,, = 1.2,. 
u,ith c depending only on oo. t. and th« param«lcrs qf T. 
Pzvof. AIl the clcments for thc proof alrcady appcar in this sui»section, espccially in 
thc proofs of ThCOl'Cln 3.7 and Lclnlna 3.5. and wc hall assume COlni)lctc falniliarity 
with the llot.ation, tCl'lniliology, a.nd estilnates given thcl'C. Denote the mmd)cr 
of cells used in the "triln and eut'" algol'ithlii for (ho), with approximation error 
e, bv ri(e). Silnila.rly, let nj(ej) l»e thc Colrcsponding lmmber of cells uscd for 
f,i (j = 0,1), again rel)reselte, l as ]'J = 0ç), iii place of Step l. çhe 
theoreln will be proved once we establish the estinmte 
(3.26) ,(e0 + el)  2 (,,0(e0) + ,l(e) ) 
for anv «0, «1 > 0. Indeed. bv COlnbilfing this inequality with the results of Theo- 
rem 3.7 (in particular, incqualitics (3.16) (3.17)), ve can see tha.t the estimate 
(3.27) n(e0 + ml)  2ç«rO,Ç TM + 2ç«r1,Ç ri :  
is true if we set e := (ç)l/r?l--1/r, j = 0, 1, where c is the constant appearing 
there. But the fact that n  n(e0 + e) and the dcfilfition of (-) imply 
TC 
c(f),  ¢+,)(f)  e0 + ci. 
Helwe, bv the defilfition of tlle êj, thc rightmost terres of this last ilmquality are 
bounded by the desired terlns on the right-hand side of ilmquality (3.25). 
I1 order to prove estimate (3.26). we only lmed to estilnate the lmmber of seg- 
lnellts  for f. First observe in Step 3 of/he algorithm that for the thresholdilg 
condition (3.6) to hold for f, with z := e0 + e, the «onditiol must also be satisfied 
for that saine cell 0 ¢ for at least one of the fJ with corresponding threshold ci 
(j = 0.1). This shows that the tree F" = F'(f,e) determilmd bv threshold e is 
contained in the union of the corresl)Olding trees F'(ff, eg) (j = 0.1). By the COll- 
struction of segments « from maxilnal chains of F'(f) il Step 4, the Seglnents for 
f are disjoint and one of the conditions (3.10)-(3.12) m,,st hold. If (3.10) or (3.11) 
holds for a segment « of f, then 0« Ib + bl > e0 + e implies the correspond- 
ing condition for at least ont of f0 (and e0) or ri (and Zl). That is, for one of 
j = 0, 1 we must have 0« Ibl > zj, a,d s fl- at let hall of the segments of f 
this condit.ion must persist for a fixed index j (j = 0, 1). The lmmber of relnlmnt 
segments (see (3.12)), on the other hand, may be estimated bv the Stilll of the 
number of remirent segments of f0 and fl, plus the lltlnlber of ?lew bralcllilg cells 
which lna3" arise within the union of the trees of f0 and ll. These new colis are 



2606 B. KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY 

introduced in F"(f, ) when two chains, exclusive to each of the F"(f j, ej), meet, 
therebv dividing the existing chains for each of the trees and creating an additional 
seglnent. It is easy to see that the nulnber of such new branching cells does hOt 
exceed nlill { rï (f°, e0), Uï (fl, gl ) }- 
This accounting of the three quali(ving conditions (3.10)-(3.12) for segments gives 
 lllgX{#(f 0 0) #(fl,l)} + lllill{Fï(f 0,0),rf(fu 1 
 #2(f °, g0) + #(fl, gl), 
which iml)lies lhe desired estimate (3.26) a.nd COlupletes the proof.  
3.2.2. The «ose 0 < p < oe. XX' now l'eturl to coml)leting Steps 3-5 in the ce 
lha.t p < oe. The arguments are quite silnilar to the case p = oe in the previous 
sui)section, and we shall use the notation there and indicate only the differences. 
hdrodu('e a new l)a.raln(,ter O, where 0 < ç < p. and fix g > 0. 

Step 3. TrblTniTg of (-)" (1 _< , _< K) witb e. This step is the saine as in Case 1 
(p = oe) wilh (3.5) l'('l)lace(l i)y 
In contrast to lhe case p = ., the error 1£ - fr I1» i 1o longer controlled soMv by 
g. It will (h'l)end Oll the smoothlmss of the flmction f that is being approxilnated 
(see Theorem a.9 bclow). 

Step 4. Partitioning the branches of each tree F" into chains and the chains into 
"segments". We proceed exactlv as iii thc case p = c, repla.cing conditions (3.10)- 

(3.12) by the following: 
(3.29) 
(3.30) 

Ibo, II01 /» > e (cae of p = 0). 

1/0 /i+p \ 1/0 
<_ e, but y'Albo, llOll/") °) > e (case of p > 0), 
j=i 

(3.31) 

% IIo.,-I'/')  _< . 

Step 5. Rewriting elements from certain segments of ". This step is exactly the 
saine as for the case p = 

•Error estimation for the "trim and eut" algorithm (case 0 < p < oc). 
Suppose that the "trim and cut" algorithm bas been applied to a function f with 
0 < c0 < p and e > O, as described above. Let ArC(f)p = eeh ceçe, A C . be 
the approximant produced by the algorithm. We denote 
(e) := #A« TC 
, A.(«)(f)p := IIf- AT C(f)»llv • 
and 
TC • TC . 
A,, (f)p := mf{A,(«)(f)p n(g) < 'r}. 



NONLINEA1R PIECEWISE POLYNOMIAL APPROXIMATION 2607 

Theorem 3.9. If f G 177(7). where c > 1/0- 1/p a,,d - -- (c + )-1. then for 
each  > O. 

(3.32) TC • 
and hence 

Proof. We first estima.te (e). 
inequality of (3.28)) in Step 3, it follows that 

From the stopl)ing tilne critel'itm (the converse 

(3.34) e < ( , (IbollOI1/p)o) /° ( \ l/r 
  (IbollOI1/P) r) (SillCe T _ O) 
\0C0 o 0C0'» 

for ea.ch 00  F Ï, whi('h ena.blcs us to repeat thc argmncnts from the proof of 
Theorcm 3.7 and obtain the estilnatc :/#Fï <_ c IlfllB7(-)- In going ful'thcr, we 
use (3.30) iii a, silnila.r fashion and the a|ove t infer as in the proof of Theorem 3.7 
that 

(3.35) 
This implies the desired estimate for t(¢). 
ATC 
It rcmains to estinlate t}lç error IIf- ¢«)(f)pllp. xx first estimate lift- fllp. 
To this end, we group the rcmoved cells into collections of comparable 7-11ornls. 
We denote by 
E" := {0  O" F" "0  0' for a.lV O'  ' Fu, O' 0} 
t.he set of all cells a.t which a trimmed branch sta«ts. Note that for each 0 
the inequality (3.28) holds. Therefore, we ca.n partition " into disjoint collections 
-j, j = 1, 2 ..... L", such that =" L =" and 
(a.a) 
0°j 0C0  
for ail j = 1, 2,..., L" except possibly for j = L ", when the leftlnost inequality 
may fail. Hence, since the cells Dom E" has-e disjoint interiors, and recalling that 
IbollO, /  HboçoHv, we obtail, 
L 
j=l 0¢E 0C0 ° -- 0°E 0C0 ° 
L 
kj=l 0*eE" 0C0":" 
L  

<_ c(-2P/OeP)/P=c(L")/Pe, 

j=l 
where we used the embedding inequality (2.26). To estimate L" we once again 
exploit the idea used in estimating :/#Fï (see (3.18) (3.19)). Since 0 < r <_ 0. we 



2608 B. IARAIVANOV, P PETRUSHEV, AND R. C. SHARPLEY 

We use this and the fact that the collections 
_j m'e disjoint to obtain 

(3.38) 

Comlfining (3.37") and (3.38), we oi,tain 

Il.f,,- ff,, IIp _< «(e-TIl¢;ll(r))l/Pe =  ""ll.ï,,ll,/lr, 

and honte i»v standard sulm&livitity estimates for Lp ((I < p < ,OE) we mav estimate 
the SlIlII 

(:.3.) IIf- frllp 

l/p* 

where 1'* := rein{ 1. t'}. 
T(, complete the proof of the the(n'em, we must estimate Ilfr - A'IIp • This 
«liffers ri'oto our earlier arguments in the case p = . which involved the error 
estimate (3.15) over a ring of a segment. For any such ring R« (a  ") we use 
instead the estimate 

where we used the embedding inequality (2.26). From the above, using that ail 
rings {R«}«e:" bave disjoint interiors, we obtain 

(3.4(}) 

lift" - A"llp -< ( ' lift,'- "11,(,,) p )/P -< 

('Olldining (3.40) and (3.35) yields 

and hence 

Ilfc --'ffc(f)pllp 

1]p* 



NONLINEAR PIECEVVISE POLYNOMIAL APPROXIMATION 2609 

where p* := lnin{ 1,p}. From this and (3.39). we oltain the api)roI)riate estilnate 
whiçh corresponds to (3.14) of the case fin p = 
4 TC f 
(3.41) IIf 
Lemlna 3.10. Let f = fo+ fl, here f = oeo boço. f = oeo ço (j = O, 1) 
1)--1 
u,itb j _ > ! o - p 1 (0 < 0 < P) ad r A := 1/(aj +  . j = O. 1 . Furthermore, s«ppose 
the "trim and eut'" algovitl«m bas been applied to f, usi9 the above represetation 
off it place of Step l, with 0 < 
Tbe we bave 
(3.42) re - 
(«+«(I) 

(3.43) u(so -[- Sl) < C S-T°JÇ ° nt- 1 "1 " 
ad. tberefore, 
Arc{t: ) < C(ll-aOJo-t-ii-cjk/'l) il = 1. o, 
(3.44) 
 J P . ..... 
where c depend o'ul9 ot p, 0, a0. 1, atd the pa'rameters of T. 
Pro@ The proof is very silnilar to the proof of Theorem 3.9. and we shall onlv 
indi«ate the differences, using thc notation and ideas from there Those differences 
are in ebtilnating Fï,  and L  (see (a.aa na (a.as)). Fom th stopping 
«riterium (converse inequality to (3.28)) in Step 3. it follows that. for 0 °  F Ï, 
o + 1 <  (IbollOI1/)  
OCO  
1 ( )1 
 +c (Ibll0ll/)  
OCO  
1/ri 

/ )l/r°/ ) 
< c o (Iblloll/p) ° ÷% (Iblloll/V) 1 
\OCO o \OCO o 

where c o := max{1.21/°-1} and we used the fart that To, T 1 <__ O" 

Therefore, for 

each 0   F Ï, at least one of 
0 < C(- (]b]]o,l/P) "rO) 
\0CO 

or 1 < ¢0  (Ibll°ll/;) 1 
\OCOO 

must hold. Denoting by FÏo and Fl the sets of all 0 °  F Ï for which the first or 
second inequality, respectively, holds, we obtain 
O°FÏ oco  



2610 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY 

and hence 
(.) # _< #0 + #> _<« (0a 0 + ï). 
 obtain silnilar (with the saine right-hmd-side quantity) for #E  and L  by 
using the saine argument. The estinm.te for #E  gives the desired estilnate for 
} 111W use estilnates (3.37) and (3.40) iii the proof of Theorem 3.9. with e = 
go + e;, together with the above estilnates for #E and L , to obtain 
(a.46) rc 
A(«+«,)(f)p 5 c(eo + ml) 
ffoin which the desired estinmtc (3.42) fl)llows. The filml estinmte (3.44) is proved 
t)v sel(,cting éj = (2('/t)l/r, which I)v out rcsult (3.43), gives that n(g0+gl)  n 
«llld Ho 
Ac(f)p  A(eo+e)(f) p  «'ll/P(gO +E1)  «" II--a°,ço+ I--al.çq • 
whore we Imve used (3A{i) in the s(,('ond in('qualit3ç 

3.3. "Push the error'" algorithm. The i(lea of this algorithnl to oilr knowledge 
first al)I)eared iii [5]. ()lit goal is to a(laI)t this algorithnl for nonlinear n-terre 
('Olll'allt ('lelllellt al)i)roxilnation iii the lllliforlll llOrlll alld perfect it so that the 
resulting algorithln achieves the rate of convergence of the best approximation. 
Iii §a.a.l, we describe the "lmsh the error'" algorithln iii its simplest and nlost 
naive forln. \Ve follow with three exalni)les which illustrate deficiencies of the 
simple algorithm and the types of traps to which it lllay fall prey. In §3.3.2, we 
give our refined version of that algorithm. Throughout this section. ,ve assunle that 
7- = l.J=0 OE,, is an LR-triangulation of solne compact polygonal dolnain E in IR 2. 
where the approximation takes place (see §2.1), and f _ C(E). 

3.3.1. A naive "'push the error'" al9oritbm (p = oc). We t)egin by outlining the 
basic clClllents of the algorithm. 
Step 1 (Decoropose). In this sut)section we denote by Qj(f) the piecewise linear 
contilmous fimction that interpolates f at the vertices I of all triangles froln . 
Clearly f  C(E) cal1 be represented as follows: 
(3.47) f = Qo(f) +  (Q(f) -Qj-l(f)) =:  co2o. 
3=1 0 
where the series converges unifornlly. Ill practice the series ternfinates at some 
finest level a (J > 1), so that 
J 

Assulning that initially f = 0ea coço, t.here exists a fast and efficient, procedure 
for obtaining (3.47). 
Step 2 ("Threshold'" and "'push the error"). Fix e > 0. We shall begin at 
the coarsest level f0 and proceed consecutively through to higher resolution levels 



NONLINEAFI PIECEWISE POLS:NOMIAL APPFIO\IMATION 2611 

O,O2 ..... (9.I- V'e define A0 as thc set of all cells 0 • (90 such that [coi > ŒE 
(ll011 = 1), and sel 
0Ao 0o 
Next we rewri[e all remainilg Ierlns coco (0    A0) at [he next finer level 
and add [he resulting terlns to the correspolding terlns froln (coo)oeo. Thus we 
obtain a represelda[ion of f iii [he forlll 
0 j=2 0 
 next process Ille CouranI elelnents at level 1. ç <lefine At as the set of all 
0  #) such that IboI > e, a.nd set A := OE0eA, bobo. AI1 lelnaining terres bobo, 
0  (-)  At, we rewrite at the finer level  and add thc resul[ing terres o the 
corresponding [erlns (coo)oeo:. The representa[ion of f at [his stage is written as 
.l 
(3.48) f = 1 + A1 +  00 +   CoCo" 
Ve COllIillllP in this way until we reach the filleSt (i.e., highest resolution) level 
j. The only lnodificatiol at this finest level is IllaI we discard all telmS whose 
coecieldS in absolute value do hot exceed our Ihreshold paralneIer e. In Ihis wav 

Since only slnall terres (lb01 _< e) at a single (in this case, finest) level m'e discarded, 
they ca.lmot stack up, and we have 
Sollle lnodifications must be lnade, however, to insure that this silnple and efficient 
algorit hln will achieve sparse l'epreseltatiols iii ail asylnptot ically optilnal sense and 
avoid hidden t raps that will result in using too llla.ny terlllS iii the approxiilmtiolL 
We indicate briefly each of the possible pitfalls to keep in nlind, before developing 
the algorithln in fldl in the lleXt, sllbsectioll. 
Trap 1. The interpolation schelne we used to l'el)l'eSeld f iii (3.47) leads to 
diflïculties, since it does hot always lead to sparse represelt.ations. "Ve give here a 
univariate exmnple which lllaV be easilv extended to two diInensions. 
Let E := [-1, 1], and let f be the hat fimction on [ ',1 2__N_]l for N sufficiently 
lin'ge, i.e., f(z = ç(2Na ") with ç(x) := (1 -Jxl)ll[_l.ll(X), x • IR. We assmne 
that 7 consists of all dyadic subintervals of [-1, 1]. Using the interpolation schelne 
described in Step 1 at the coarsest level, we nmst interpolate the extremes at 
-1,0, 1 in order to decrease the L  error. The resulting error after this stage, 
however, is 1 - --v. Proceeding with the haire "push-the-error" algorithm with any 
 < ½ results iii ail index set A with #A  N. However, the best approximation 
is achieved using the single fine scale element ç(2Nx). Therefore, any reasonable 
algorithln that retains  terres in the approximant should give a rate of COllvergellCe 
O( -'r) for any 3' > 0. 



2612 B. KAIRAIVANOV, P. PETIRUSHEV, AND R. C. SHARPLE st 

Trap 2. For a given ¢ > 0 the algorithm as currently described may produce a 
great lmlnber of undesired terlns due to the superposition of a large number of fine 
level nonintersecting terlnS (coço) with a singlc coarse level terre 
(3.49) f=e([-1.l]+o). 
 set  a.s a set of disjoint cells 0 rioto level 2 with 0 C (-5. 5), where 5 = 2 -'. 
It is clear that we tan choose thesc cells for , so that «M = 2 N. At the central 
vertex x0 of each cell 0 we have f(.ro) > e(1 - 5) + Se = e. The "push-the-error 
algorithm'" will pro(luce an ilmcient apl)l'oxinm(iolL since it will hot select the 
('oarse fil'St terre in (3.49) as one might hope. Instead, no such element will be 
('hosen at the ('oarsest h, vel, and thc errol will be pushed. At each successive 
stage the co«'c'ients of the rewritt(,n des('endan Couralt elements for 00 will ail 
again lie I»encath thc threshold and be furIhor rewritlen until all cells are on level 
'2 N At that stage they will be combined with the relnaining terlns in (3.49). 
The corr(,sp¢mdilg ce,Ils will now bave coccieltS that exceed the threshold and 
nmst I)e s¢'l¢'('tcd. In'Oducing at h,ast 2 N Wrms in the api)roximant. As indicated 
al)ov(', a d('sirablc algoriIhln should haro anticipatcd the trap of lllanv sma.ll, finely 
suI)i)Olted clcmcnts Ihat may corne, ai a laie stage, and would have chosen for this 
flmcIion the aI)l)roximaIion (with thrcshokl e) that consists of a single elenmnt, 
namely e ç[-1,1l- 
ap 3. The final examplc is onc that outmalmuvers a quick remedy to Trap 2, 
i.e., meroly thresholding ail snmll terres at Ihe finest level. For a given e > 0. we 
defilm 
f= e [-.11 + Sj ç[0.z-l + eç[0,Z-MI , 
where mi = j2, 5j = 2-1e ' and BI = 2 N. In this example, elelnents are again 
building near the origin, but now aI)pear at lllany levels with small alnplitudes. 
The "'push-the-crror" algorithln will again take no elements at the coarsest level 
and push the error to the llPXt lcvel. Coll[illtling with the algorithln, we are forced 
to take essentiallv ail tOllllS as the approxilnation to the given fllllCtiOll when. 
optimally, only two terres lmed be takcn. 
It is obvious that we can take each of these telnplate exalnples as building blocks 
and i)uild flmctions to cause these problems for ail «, at ail locations and scales. 

3.3.2. "'Push the error" algoritlm in the uTiiform orm (p = oc). In this section 
we indicate the refinements lmeded in order to gua.rantee that the "'push the error'" 
algorithm will achieve optimal rates of approxinmtion. As with the "'triln and eut'" 
algorithm, we break it down into lnanageable steps. 

• Description of the algorithm. 

Step 1 (Decompose). For f E C(E) initiallv represented by (3.1), we lllcy assume, 
without loss of generality, that there exists a finest level )j (J > 0) such that f is 
written as 

(3.5o) 



NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2613 

Step 2 ( "'Prune the shr'ubs"). In the current algorithm we are hot able to organize 
the cells of O into trees as we did in the "trim and eut" method, since, once we 
rewrite the error on a finer level, a.djacent trees are immediatelv affected and we lose 
the benefit of their intended organization properties. This step of our algorithm, 
however, is analogous to Step 3 of the «trim and eut" algorithln.  fix e > 0 
and let e* := e/2. Oto" goal is, by discarding small insignificant terres boço in 
the representation of f ri'oto (3.50), to prevent our refined algorithm from heing 
trapped by a situation such as that described in "Trap 2"' (sec the haire "Imsh 
the error'" algorithm of 3.3.1). b shall remove such terlns, but insure that the 
resulting uniform error is at most ê* and dcnote by F the set of ail retained cells. 
Iu addition, we shall construct a set F I C I', consisting of "final cells'" in F. 
First, we lmed to introduce ail organizational concept as a replacemcnt for the 
tree structures of 3.2.  shall say (figuratively) that a cell 0 G  sits on anothcr 
cell 0  G (-), if 0 is at least as fine as 0  and its interior (dcnotod by 0 ) intcrseçts 
the interior of 0% Fnrtherlnorc, tbr 0  G (-), we denote the collection of all cells that 
sit on 0  by 
(3.51) Y0* := {0 e O: 0   0 ° ¢ 0 and level(0) k lew'l(0°)}. 
The procedure of Step 2 will bcgin at the fincst level and proceed to the coarsest, 
level by level, coustructing sets F I aud I'. To initialize the procedure we Imt into 
F f ail significant cells 0 G Oj, i.e., such that bol > e*.  place in F anv cell ri'oto 
Oj that sits on a cell from F 
The inductive step proceeds as follows. Suppose that all cells from ) with lcvels 
j > m (0  m < .1) have already been processed.  now describe how to process 
,.  place into F I all cells 0   (-), that satisfv 
(3.52)  Ibol > *, 
and for which there is no 0 G r I from a higher level (i.e., > m) that sits on 0*. A 
cell 0* rioto 
We mv consider the current version of r f as an intermediate (m-th) version of a 
final set for F. Obviously, a cell 0* from O is discarded and hot placed in F if 
(3.53)  Ibol  *, 
and there is no 0 
The procedure is ternfinated after O0 is processed and Step 2 of the algorithm 
is completed. 

The two sets of cells F and Fy (F/ C F C (r)) produced bv Step 2 have the 
following properties, which follow directly from their construction: 
(i) if 01.02 • F I and level(01) ¢- level(0.2), then 0ï ç?0_ = 0: 
(ii) for each 0 ° • F I, the inequality (3.52) holds: 
(iii) for each 0 ° • F, there exists 0 • FI that sits on 0 °. 
Vre set .ff := 0er boço and define 
{ bo, if0•F, 
(3.54) ao := 0, if 0 • 6) \ F: 



2614 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY 

then obviously 
(3.55) fr = Z aç. 
It follows fronl the construction that 

(3.56) [If- frll - ¢*- 
Indeed, to see that this estinmte holds, we let D denote the set of all cells 0 • 6) 
that were (liscar(l('(1 (luring the iml)h'lnentation of Step 2. i.e., T) = 6) \ ['. Let 
.r • E |)e arl)itrary. If .r  [,-J0ez) 0, then x does hot belong to anv cell tllat was 
(liscar(h,(l, and so .fr(x) = f(,r). On the other hand. if x (5 [,-J0ez)0. then tllere 
exists a oeil 0  (5 T tlat contains .r and has coarsest level. Since 0 ° ,cas discarded. 

the in('(luality (3.53) lllllst h()hl. I! follows that 
If(x)- fr(x)l = I Z bo?o(x)l <_ 
OD 

OEYoo 

wh(,r(, wc have l(,rlnaliz(',l our (,h,m(,nts s(, that ]]ç0]] = 1. This verifies the desired 
in('(lua]ity (3-56). 

Stel) 3 (Pu,s'h the error). \Ve n()w l)ro('ess cells of fr with œe*, starting fronl the 
coars('st level (-)0 and continuing to fin('r levels. The outcome of this step will be 
an al)t)roxinlant A := A(f) of the fi)rm 

(3.57) 

J J 
3=0 3----00GA. 

where A./ C (-) and Ai will det)end on f. 
\\e use the notation 

.to := {0 (5 O-0 ° çl0 ° ¢ 0 and level(0) = level(0°)} 

for cells fronl the saine level as 0 ° which are adjacent to it. 
We start fronl the representation of ff in (3.55). We define --\o as the set of all 
0 • Oo such that f01 > e* (ll011 -- 1), and ve set -\o := [,-J0eÀo ,U0. \Ve denote 

.40 := Z aoço =: Z doço. 
0Ao 0A0 

For each 0 ° • 6)j, ç0o call be represented as a linear combination of ço'S with 
0 • 6)j+l- \Ve use this to rewrite (represent) all renmining terres aoço, 0 • 6)0 \A0, 
at the next level and add the resulting ternls to the corresponding terres aoço, 
0 • 6)1- We denote by do¢?o, 0 • 6)1, the new ternls, and therefore obtain a 
representation of f in tlle forln 

f = A0 -I- 

J 
OG j2 0 

Continuing with the next level, we define / as the set of all 0 • 6)1 such that 
Idol > :, set Al := [,JoeA œeo , and define ,Al :--- 0eh do¢?o. As for the previous 
level, we rewrite the renmining ternls doc2o, 0 • (01 \ Ai, at the next level and 



NONLINEAR PIECEWISE POLh NOMIAL APPROXIMATION 2615 

add the resulting terres to the corresponding t.erms aoço, 0 c:_ t92. We obtain the 
following representation of f: 

f = No + A + E 
dodo 
+ 

J 
j=3 0E(gj 

We contilme in this wav tmtil we reach the highest level of cells 19j. At level 19j, 
we define Àj, A j, and `4.j as al}ove and discard all terres doço, 0 e_ (-)j \ Aj. 
We finally obtain our aI»proxinmnt `4 = `4P(f) in the form (3.57). We denote 
J 
A := A« := l J j=0 Ai and À := À := LJg0 Àg, and so `4 = Y'-0e^ doço. 
Since we throw away only elements doç)o with Idol <_ e* at the finest level 19j, 
we have the estimate 

Hic - `411 
O(-) j \ A j 

and hence, using (3.56), 
(3.58) 

This completes Step 3 and with that the description of the algorithm. 
We want to point out an important distinction between the "push the error'" 
steps in the above algorithm and the "naive" algorithm described in §3.3.1. The 
difference is that each time we put a significant terre doço (Id01 > e*) into ,4 we also 
include the neighboring terres (i.e., from the index collection .Vo). This prevents 
our algorithm from being defeated bv a situation like that described in "Trap 3"" in 
§3.3.1. 
• Error estimation for the "push the error" algorithm. Suppose "push 
the error" is applied to a function f with  > 0. and `4(f) is the approxinant 
obtained: `4ff(f) := -'0e^, doço. As in the "trim and eut" method, we use the 
corresponding notation 
n(e) := #A«, P P 

and 

P(f) := P(f,T) := inf{P()(f) "n(e) <_ n}. 
We reinark that if f 6 B(T), then by the Embedding Theorein 2.7 it follows that 
f is continuous. Estimates (3.59) and (3.60). established in the following theorem, 
imply uniform convergence of the "'push the error'" approximants to f and provide 
the necessary rates of approximation bv the method. 

Theorem 3.11. If f c:_ BT(T ),  >_ 1, r := 1/c. then for each e > O. 
(3.59) P  
A(e)(f) _< e and ri(c) < ce-HflIB7(oE), 
where c = 6N. Furthermore, we bave 
(3.60) AP(f) < c-llfllB(» . = 1,2 .... 

with c = (6N O). 



2616 B KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY 

Pro@ In order to prove (3.59), we first observe that the direct approxilnation 
estimate ,&/x P 
,(«)(f) <_ e follows from inequality (3.58) in the constructioll of the 
algorithm. Therefore it only renlains to show that #A_ _< ce-TIIfl[).(7). Clearly, 
(3.61) A« _< (N0 + 1)(#À), 
and we need only estimate the cardinality of  := . We split À into two disjoint 
sers,/ï'I and/ï,,.. \Ve define/ï,i as the set of all final cells iii A. that is. the set of ail 
0 Ç/ï, for which there is 11o 0' Ç/ï, of a higher level sitting on 0. We set .; := 

\Ve shall lnake l'eI)eated use of the fi»llowing simple lenmla. 
Lelnnla 3.12. Suppose Ad C (-) satisfies the condition that cells front different 
let,els do ot hat,e interiors that intersect. Then each 0 Ç (-) may sit on al most 
No + 1 oeils from . 
Pro@ The silnp]e hyp(»thcsis of thc lonlnla just states that for a cell 09. to sit on a 
«cil 0, it lnust be on the saine lcvel: but lhere can be at most N0 + 1 such cells. [] 

We first estilnalc the nuln|mr o[ clelnents Ç/ that arise as final cells in Step 2. 
r each 0 ° Ç 1"i, we have, by (a.52), 
(3.6oe) e* <  Ibol  (  Ibol) '/r (  ). 
OE'oo 0E'0o 
Clearly. FI satisfies the hypolhesis of Lenmm 3.12 (see Property (i) of FI, which 
is stated following (3.52)). and hence each 0 Ç (-) mv sit on at most  + 1 cells 
from Fi. Using this together with (3.62). we obtain 
llfll(oE) :=  lbol   ( + 1) -   lbof  (. + 1)-(#Fi)(e*) . 
00 O'»F I 0'0o 

which, since "r _< 1. ilnplies 

(3.63) #I'i < 2(N0 + 1)e-rllf]lv(7). 
We next estilnate #]OEI" the lmlnber of final cells for the index t Ç constructed 
in Step 3. Clearly Donl that construction, a ccll 0 Ç Ç lnaV occur onlv if 0 Ç F, and 
hence  C F. On the other hand, from Step 2, for each 0 Ç F there exists 0' Ç F 
sitting on 0. Therefore, for each 0 Ç I there exists 0' Ç F I sitting on 0. But 
satisfies the hypothesis of Lemlna a.12 (with ,M replaced bv I), and hence a cell 
0 Ç F I lnW sit on at lnoSt  + 1 cells froln I- Dom this and (3.63), we have 
To colnplete the estimate for , we must estilnate .. Suppose 0 °  Ç := 
  I, and let 0'  À be a cell sitting on 0 ° with level(0') > level(0 °) md such 
that level(0 ) is the minilnUln of the levels of all cells in  sitting on 0 °. Such a 
cell exists, by the definition of À, but it is possibly hot Ulfique. XX denote by 
Zo the set of ail 0 Ç F which, while "pushing the error" from 0 ° in Step 3. have 
contributed to the terre do, ço,. Due to the lninilnality of 0'. we see that 
(3.65) do, = do,o,(vo,) =  
OE Zoo 



NONLINEAR PIECEV¢ISE POLYNOMIAL APPROXIMATION 2617 

where t'0, is the "central vel'tex'" of 0'. 
using (3.65), 

Since 0' E À, then Ido, I > e*, and hence, 

(3.66) * < Ido, I <_ Il  bovo,l <_ (r  1). 
0Gîoo 
It is easilv seen that ea«ll 0 G Zo satisfies the folloving properties: 
(a) 0  0', 
(b) level(0 ) < level(0)  level(O'), 
(c) the "central vertex" of 0 lies on 0 °, and hence 0 sits on 
Property (a) follows by ohserving that the support of an elelnent which is rewritten 
at a finer level always «ontains the supports of the COltritmting fille" elelnents. 
Property (b) holds, since . ç A, and hence 11o terres boço with level(0)  level(0 °) 
lllay contribute to do, ço,. Note that it is possible that thel'e are 0 that salisfy 
properties (a)-(c) above but do not belong to 0.. 
Next. we show that each 0 G [" may belong to at lnOSt  + I sets Z0* with 
0*  À,.. Ind«ed. let O  F and sui»pose 0 °  .,. is such that 0  Zoo. In 
the following, we shall use the notation ri'oin al»ove that inw)lves 0 c, but we will 
consider su«h 0  as arbitrarv in . Let .Mo denote the set of ail 0  Ç  such 
that 0 G Zo,. In particular, 0 ° G Mo- k, fix Mo and shov that it satisfies the 
hypothesis of Lemlna 3.12. Indeed. let 0, 02 G -0 from different levels. But this 
ilnplies 0 G Zoj (j = 1,2), and we lllV as well consider 0 = 0 ° and sav 02 = 0 , 
where level(0 ") ¢ level(0). Evidently. level(0 ") < level(O'), ri'oto property (b) 
applied to 0" and 0. 
Bv sylnlnetry, we nlay assume level(0 ") < level(0°). If (0)   (0°)  ¢ 0, thon 
0 ° sits on 0  and hence, since level(0) > level(0°), 0 Callllot be iii Z0, which is 
a contradiction. Therefore. (0)   (0°)  = 0, which verifies the hypothesis of 
Lelnma 3.12. 
Now that Lelnnla 3.12 can be applied to M0, then 0 ( anv other cell froln 
ln<V sit on at most N0 + 1 cells 0* G «0- Therefore, 0 lnav belong to at most 
X + 1 such sers Zo* with 0*  ,.. Using this and (3.66), we obtain 
I[/[IBe()   lbol   (So + )-1   IboV  (; + 1) -l(Aï)(- 
OGF OO,. OGZoo 
Therefore. it follows (recall that ç < 1) that 
I combine this estilnate with (3.61) and (3.64) to obtain the desired estimate of 
#A« in (3.59). Estilnate (3.60) follows ilmnediatelv ri'oin (3.59). 
The following lemlna will be needed iii 5. 
Lemma 3.13. Let f= f0+fl, where f= oeoboço, f = oeobôço (j = 0.1), 
,d  =  +  (u 0 e ), d 
 := Ib$1" < , (j = 0.1), 
where oo. al  1 ad r0 := l/a0, ri := 1/al. Furthermore, suppose that "'push 
the evror'" is applied using the above repreentatio of f , with  := 0 + 1. where 

OGZoo 0 o 



2618 B KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY 

£0, £1 > O. Tben we bave 
(3.67) 

Aff(o+)(f) _< eo + 1, 

1(£0 q- £1) -- OEr°-A/' -° -[- C-'/'IH; 1, 

Therefore, 

#à; < (No + 1)(#r;) < (No + 1)(#Fï + #F}) 
(3.72) < 2(N0+ 
To complete the proof, we must next estilnate #Àr. For each 0 ° G /r, we define 
0' G/ and Zoo exactly as in the proof of Theorem 3.11. Similarly as in (3.66), we 

and hence ( rj <_ I) 

:/#F _< 2(5o + 1 
)e A.. 

(3.6s) 
wheve c = 6N. Co.nsequently. 
A, (f) < co-«°JV'o + CI'--alI, "11 = 1.2 .... , 
(3.69) P , « = 
Pro@  follow in the footstct)s of the proof of Theorem 3.11.  shall use the 
notation from there, and only indicate the differences as they arise.  denote 
e* := e + e with Q := e/2, j = 0, 1. Estimatc (3.67) is immediate from the 
description of the algorithm. 
It remains to provide estimate (3.68) for the mlmber of terres used in the ap- 
proximation. As iii (3.61), we have 
(3.70) "(g0 +g) := #A 5 (No + 
where we denotc À := , and ÀI and À,. bave the saine definitions, proceeding 
exactly  in lhc i)roof of Theorcm 3.11. Çonfimling as there, we have to estimale 
#F I. For each o  FI" we have, by (3.52) and lhe fact thal 0 < ri  1 (j = 0.1), 
that 
0EY0o OEY0o 0EY0o 
 (  b[r°) 1/r° +(  [b[T1) l/ri" 
OEYoo OEYoo 
Dom lhis, it follows/ha.t, for ea«h 0 ° E F S, al let one of 
(3.n) 4 <(  Ibgl) '/° or «1 < (  Iblrl) 1Ci 
OYoo OEYo 
must hold.  dcnote by Fï and F} tlle sers of ail 0 ° E F S such that the respective 
condition from (.71) hohls fi»r either j = 0 or j = 1. For j = 0.1. we have similarly, 
 in tire proof of Theorem 3.11. 
 :=  ]b]  2 (N0 + 1) -1   ]bl  2 (N0 + 1)-l(F})(«ff , 
0E 0OEF} 0EYoo 



NONLINEAR PIECEISE POLYNOMIAI. APPROXIMATION 2619 

have 

or 
(3.74) 

 <( Z Iblrl)llrl 
OEZoo 
nmst hold. \Ve dcnote by j,0. and ,[ tlw sers of ail 0   ,. fir whi«h (3.73) and 
(3.74) hokl. rcspe«tively. As in the proof of Thcorcnl 3.11. each 0  (-) may belong 
to at lnOSt  + 1 sers Z0. 0   ,.. Thcrefore, for j = 0.1, 
.¢  lbDI , (No+ 1) -1   Ib$1 '  (q, + 1)-1(#)(¢) r', 

and hence 

Therefore, 

#X{ _< 2(No + 1)e}-rflÇ'j , j =o. 1. 

f 
_< #j,o. + #X,I. _< 2(No + 1)tero.N'oO ..11_ e-TiHïl) " 
This estimate, together with (3.70) and (3.72), implies (3.68) (sin«e No > 3). Esti- 
maie (3.69) follows by usi,lg e := (2c)",,,-"'A/} (j = 0, 1) i,1 (a.cr) a,lO (a.s) to 
obtain n(eo + 51) __ '/l, and so AP(f)OE _< APn(eO+el)(f)oo <__ gO + gl. [] 

4. BEST ?t-TERM COURANT ELEMENT APPROXIMATION 
In this section, we &SSllme that 7- is a locally regular triangulation of a bounded 
polygonal donlain E with parameters ]V 0,/[o, r, p, 6. and :0 (sec §2.1). We denote 
by (I)7 the collection of all Courant elemcnts ç0 generated by 7-. Notice that 
is not a basis; (I)T is redundant. \Ve consider nonlinear n-terre api)roxilnation in 
Lv(E ) (0 < p <_ oc) from 7, where we identifv L(E) as C(E). Ollr main goal 
is to characterize the approxinlation spaces generated bv this approxilnation, with 
emphasis on the case p = oc. We let E,(7-) denote the nonlinear set. consisting of 
all continuous piecewise linear functions ç of the fornl 

' =- Z a0990" 

06.A4 
where .M C 0(7-), #.M <_ n and .Ad may vary with S. 
the best Lp-approxinlation of f  Lp(E) frolll 
an(f, 7)p := inf IIf- Slip. 
s.(7) 
In or(Dr to characterize the approxillation spaces generated by (a,(f,T)), we 
begin in this section by first proving a compalfion pair of Jackson and Bernstein 



2620 B. KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY 

inequalities, and then follow with the usual teclmiques of interpolation of operators 
(see fol" exalnple [6], [15], [13]). 
111 the following, we aSSlllllO iii general that 0 < p 
p =  and a > 0 if p < : in eithcr case we set 1/r := a + 1/p. 
Tlleorem 4.1 (Jackson estinmte). If f  B(ç). then 
where c dçperMs orly on a. p and [he paromelers of T. 
Proof. Estimate (4.1) follows ffoin anv of our constructive algoritlnns as formulated 
in the corresponding Th('orelnS 3.1, 3.7. 3.9. or 3.11. 
Theorenl 4.2 (Berlsteil estimate). If,S'  E(T), then 
(42) IISIIB(r)  
w5ere c deperds only on «. p, and toc parameters of T. 
l'roof.  shall piove estimatc (4.2) only in the case p = OE. For the proof when 
l' < OE, sec [11]. SllppOSe ,%"  Z(ç) alld ,%' =: E0M COCO, where 
and # 5 n. Let A l»e thc set of ail triangles   ç that are involved in ail 
relis 0  cM. Then N = aeA 5"5, wllerc N5 =:  • Ps. P a linear polynoinial. 
Evidently, #A  V o #cM  c. 
k s[lall utilize the natural tl'ce structurc in ç induced bv the inclusion relation: 
Each trimlglc   ŒEn bas (COlltaills)  .1[ 0 children iii ŒEn+l alld Olle parent in 
ŒE,,-1, ifm k 1. Let F0 bethesetofallçsuch tllat 5D5' %rsomeS'A. 
We denote bv I'b the set of ail braiching triagles iii F 0 (triangles with more than 
one child in Ç0) and by I- the set of ail children of branching triangles in ç (which 
lnav or lllav not belollg to Y0)- Now, we eXtPlld F 0 to Ç := F 0 U Ç.  also extend 
A to  := A U Fb U F. hl addition, we introduce the %llowing subsets of F: the 
set Ff of ail filtal triatglcs in Ç (triangles iii Ç COlltaillillg 11o other triangles in 
I'), the set (i'0)y of the final trimlgles in F0. and the set Fch := F  [k of ail chain 
tciatg[es. Note that each triangle 5  Fch has exactlv Olle child in F.  lllaV 
argue as we did %r ri'ces of cells iii (3.9) that the lmmber of branching triangles 
does llOt exceed ttle llllnl]l,r of filial triallgles, #l'b  #(F0)y, alld SillC (F0)y C A. 
t]lell #Çb  Cil. Using tllis, we llave #F 5 6 #rb  cil, #F I  #A + #F  ch, 
and #  #A + #Fb + #F  cl. Içeep in mind. llowever, that #œch can be nmch 
larger thml o. 
kk lleXt est.ilnate Isla() := Ee Il-la(s);, where T := 1/ (see (2.5) 
for the notation). We dellote, for m k 0, S,, := 0e.ee(0) Coço.  shall 
use that. for   
(4.3) S(S)ŒEE = 5(S - Sm)r  IlS- 
.nd, lso, (S)  IISII()- Recall that  is the UlliOn of the collection of ail 
triangles froln the saine level as  and which share a vertex. e denote 
m0 
Evidently, #  3N0 #À  ('* (the valence of each vertex is  5). XXb çonsider 
two possibilities for each &  T: (a) &  , or (b) & 



NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2621 

(a) If A G H,,, then IL,, D A' for some A' G À ç T,. Using (2.3). we obtain 
Therefore, by summing ovcr ail m > O, we obtain in this case 

&EH rn_>0 AE H, 

m>O 

= cllSll # < c,llSll. 

 l,xl-la,,(S); <_«llSlloE  IAI/IzXl • 
Sulnnfing over nz 2 0 in this case as well. we find that 
 cllSllŒEE,   I'1/11 
OE, 
(.5) s «llSll   s «llSll #  «llSll, 
,X j=u 
where we bave once swit.ched the ordet of summation and used that I'1  11 if 
' is a child of  (sec (2.2)). 
Çon,binin inefllities (4.) and (.5), we obt.ain ISlS,  «llSll., hi«h i 
equiva}ent to (.2). 

where we used that I1',,,11,,(,) <- I1',,,11(< <_ cll*',ll(,\» <_ cllSll, 
applying Lemma 9.1. From the al)ove, it follows that 

and hence 

l<_a<_n,, A) E F«h rn'ir 

IlS- S.llL(.,» = IlS- SllL(» 
< cll(llSIl ÷ II/l/&)) < l.XlllSIIo, 

Note that if/.X_/ E "T, \ F, then Sl,, = S,,l,, and hence IlS - Srll.«,» = 0. 
Suppose Ai E Fc çl OE,,. For each A E Fco, we shall denote bv , (/ # A) the 
unique largest triangle of/ contained in A. Clearly, we have SI,x,\ ' - S,l/x¢\ ' - 
llA\k, "P"b and Sm]A, = llA, -P,, where P_x, is a linear polynonfial. Therefore. 

s(s); = s(s- s,,); <_  IlS- 
j=l 

(b) Let z_X 
j = 1 ..... .x, with ., _< 3N0. We have. using (4.3), 



2622 B. KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY 

We define the approximation space A'(Lp :: Aq(Lp7 -) generated bv the n- 
terre Courant element approximation to be the set of ail fimetions f Ç Lu(E) such 
that 
(4.6) IIf[IA(L,) := IIfllp + k  (nw«(f , T)P) q < OE' 
with the usual modification when q = oe. 
For a fixed LR-triangulation T, we dcnote bv ff(f,t) := ff(f.t:Lp.B(T)) 
the K-funçtional as defined in (2.30). The .]ackon and Bernstein estimates from 
Theorem 4.1 and Theorcm 4.2 yield (see, e.g., Theorem 3.16 of [15] and its proof) 
the following direct and imrse estimates: 

(a.7) 
(4.8) 

ct,(f, T)p < çlç(f , ,,-) 

+ Q_l.(kcr,.(f.T)p) p'_ . p* := min{p, 1}, 

wh('re c d(,pcnds only on a, p. and thc paramcters of 7-. 
The following charaelerization of the approximation spaces 4'(Lp, T) is imme- 
diate from the inequalities (4.7) and (4.8), using the ot)servation (2.31): 
Theoreln 4.a. If o < 7 < a and 0 < q  , then 
A;(,ç) = (,u(ç)), 
with equivalent norms. 
The next result establishes an important (continuous) embedding, which will be 
needed in 5 in order to identiN the approximation spaces (the ones determined by 
the algorithms, as well a.s best n-terre Courant element approximation)  B-spaces. 
Theorem 4.4. Suppose out standin9 assumptios hold. i.e., a > 1 if p = OE. and 
a >0 ifp< . gwelet 1/r := a+ 1/p, thenA(Lp, T) cBT(T ) and 
(4.9) Ilfll(oE)  clIfIIA(L,OE), 
where c depe, ds only on . p. açd the pameters of T. 
Proof.  shall prove (4.9) only in the ce p = oe, proceeding similarly as in [7]. 
For a proofin the ce 0 < p < , see [3]. Suppose f Ç A(L,T), and let 
S Ç Z(T) be such that 
(.10) IIf- sll 5 oe(f. ). 
Since a(f.T)  O, we hae f = S1 + =1(52- - S2--a) with the series con- 
verging uniforml% and hence (ç < 1) 
(.11) Ilfll()  IlSlll() +  II& - &-ll(). 
 apply the Bernstein estimate from Theorem 4.2 to S - S- Ç E2.+l (T) to 
obtain 
Il&- - &-ll()  «oell&  - -111 5 «2((f, ). + _(f,)) 



NONLINEAR PIECEWISE POLYNOMIAL APPIOXIMATION 2623 

and sinfilarly 
IISll( _< c(ll/ll ÷ Ol(f. 
Substituting the above in (4.11), we find that 

5. APPROXIMATION SPACES FOR ALGORITHMS 
Our goal iii this section is to show that the algorithlns that we developed and 
explored in §3 acllieve (il a certain sense) the rate of convergence of the best t- 
terln Courant elellellt approxinlation.  shall utilize the characterization of the 
approxilnation spaces 
Aq (Lp, 
fronl the previous section (ste TlleorelnS 4.3 and 4.4). k" shall denote bv 
A(Lv, T;Nr), .4 (Lv, T:Nrc), and Aï(Lv, T: N) t h« approxilnation spaccs gcn- 
erated bv thc "threshold", "trim and eut", and "'push the crror" algorithms, re- 
spectivcly. Nalnely, f  Aq(Lv, T; N), whcre N is N , A rc or N , if f  Lv(E ) 
alld 
with the usual lnodification wllen q = OE (it is hot quite a norm). 
Theorem 5.1. Let T be an LR-triazgulation of a botuded polggonal domain E 
2. 
(a) If p = OE,,  > 1, and  := 1/a. tben 
(5.) .4(L,ç; ) =.4(L,ç: 
with equivalent "'norms". 
(b) If O < p < oe,  > O. a.d  := (o + l/p) -, then 
(5.2) .«7(L. ç: vc)= «(L. ç:v) 
with equivalent "orms". were "trim and cut'" is applied with parameter   ç < p. 
A Tct ç and A(L; A) 
Pro@ (a) Let p = .  let A(f) denote 
denote the approximation space generated by the corresponding algorithm. Suppose 
Ilfll(;) < . Eie,ltly. «,(f)  (f), ,,d h,l«. ,i,lg Th,-m 4.4. 
It renlains to show that if ][f]] < oe, then 
(5.3) lfll(;) 
For the proof of this estimate, we shall employ Lemmas 3.8 and 3.13. Since thev 
are identical, it does hot matter if we prove (5.3) for "push the errof or for "trim 
and cut". 
Suppose f = ¢ beçe is the representation of f tllat is used wllile "push tlle 
error" or "trim and cuF' is applied. We ha.ve 
Ilfll := ( IV) 1/.  := /,  > 1. 



2624 13. KAAIVANOV, P. PETUSHEV, AND . C. SHARPLEY 

Next, we use a well-known interpolatiol technique. We choose o0. cq. fo, and r as 
follows: 1 = ni < c < o0 and T0 := l/a0, T1 := 1Cl. Hence 0 < r0 < r < ri = 1. 
Now let (Ibo 1)=1 Se the decreasing rearrangenwnt of the sequence (Ibol)oeo, i.e., 
indexed so that 

(5.4) 

lb0,1 > lb021 >--- 

\Ve fix oe >_ 0, and dcm,te f0 := =1 boo and f := Ej=2,,+I boc2o. In going 
further, we apply Lemma 3.8 or Lelmna 3.13 to f = fo+ fl, froln above, to obtain 

2 v 

+«2 -  I/%1- 
j=2"+l 

Using propert.v (5.4) and the facts that r = l/c, I < c < c0, and r0 = l/a0, we 
infer 

where we used the well-known Hardy inequalities, nalnelv, we applied the inequality 
froln Lellnlm 3.10 in [15] to estimatc the first sure and Lelmna 3.4 from [6] to the 
second terre. 
(b) For 0 < p < c, the proof of (5.2) is Sillfilar to the proof of (5.1). The ollly 
diffcrence is that file approlniatc roles of Lclmnas 3.8 or 3.13 are noxv played by 
Lemmas 3.2 or 3.10. We omit the details. [] 

6. CONCLUDING REMARKS 

Our t)rinmry goal iii the I)resent article is to quantify the nonlinear 1i-terlll 
proxilnation ri'oin Courant elements and use it to develop algorithms capable of 
achieving the rate of the best approxilnatiol. This is closelv related to the funda- 
mental question in nOlflinear approxilnation of how to lneasure the smoothlmsS of 
the fimctions. As we show in this article, for n-tcrln Courant elelnent approxilna- 
tion when the triangulation 7- is fixed, it is natural to lneasure the SllloothlleSS via 
the scale of the B-spaces B (7-). The use of these spaces allows one to characterize 
the approximatiol spaces for ally rate of convergence  > 0. It also enables us to 
develop algorithms which attain the rate of the best approximation. 
It is natural to add another degree of nolflinearity to the approxilnation bv 
allowing the triangulation 7- to var5". Thus a flmction f should be considered 
smooth of order c > 0 if infT-II/llB() < ec, where the infinmll is taken over all 
LR-triangulations 7- (with fixed paralneters). Therefore the rate of n-terre Courant 
elelnent approximation to f is roughly 0(-). Sumlnarizing, our approximation 
schelne proceeds as follows: (i) for a given filllction f. find a triangulation 7- and 



NONLINEAFI PIECEWlSE POLYNOMIAL APPROXIMATION 2625 

a B-space B'(Tf) in which f exhibits the lnOSt Sluoothness, (il) find an optimal 
represeutation of f in t.erms of Courant elements from ç, and (iii) run an algorithm 
that achieves the rate of the best tt-terul CHrant elelnent approximation. The 
first step in this schelne is the most COlnplicated one.  do not have an efficient 
solution for this as yet. Iu the silnpler case of nonliuear approxilnation by piecewise 
polynonlials over dvadic partitions, this problenl, hmvever, has a conlplete and 
efficient solution [14]. As we show, once the triangulation ç is determined, the 
relnaining txvo steps are now well understood and havo efficient solutions in both 
theoretical and practical senses. 
The three algorithlns that ve develop and explore in this article provide solu- 
tions of the probleln under appropriate conditious. A colnmou feature of these 
algorithms is the first step, a nontrivial deconlposition flOlU the redundmlt collec- 
tion of ail Courant elellents from 7. After this initial step, however, thev take 
three different routes. The "threshold" algoritlnn is COnlpletely unstru«tured ]rot 
e3" to implelnent. The drawback of this proçedure is tlmt it is uot valid in the 
case of the Ulfitbrnl nornl, and as a consequeuce it does hot perform well in L v for 
p large. The "t.riln and cut" algorithln is wdid for L v, 0 < p  OE, but it is over- 
structurcd and as a rcsult the tel'tbrnmnce suflk, rs. The "Imsh the error" algorithnl 
appears to be the preferred Ul»l»roxilna.tion nlethod. 
The algorithnls that we dew,lop in this m'ticle are hot restricted to t-terln 
Çourant elellellt approxinlation. Thev can be applied ilmnediatelv to the approxi- 
mat ion from (discont iuuous) piecewise al»proximat iou over lmflt i level t riaugula.t ions 
(for the precise setting, sec [11]). Iu this case the role of the B-spaces B7 (T) should 
be playcd by thc skinny B-spuces (T), introduced in (2.37). The results are sim- 
ilar, but sinlplify considerably. We olnit the details. 
Furtherlnore, these algorithnls Call easily be adapted t.o nonlinear -terln approx- 
imation by Slnooth piecewise polynoluial basis flulctions such  those considered 
in [3] and, in particular, by box splilleS. The main diflçrence would be that one 
should use the corresponding B-spaces, developed in [3], but proceeding in a siulilar 
malmer to this paper. 
It is natural to use (wavelet or prewavelet) bases in nonlinear approxinlatiou, and 
specifically for approxinlation in Lp (1 < p < ).  are hot aware of COlnpactly 
supported wavelets {prewavelets) generated bv Çourant elements or smoother piece- 
wise polynolnials on gelmral nlultilevel triangulations. It is clear to us that such 
wavelet bses vould he verv "exponsive'" to construct and hence are of linlited prac- 
tical value. However, iii the case of uniform triangulations, COlnpactly support.ed 
prewavelets and wavelet flalnes generated by Çourant elellents, or box splilleS, do 
exist, and have been implemented in practice. Obviously, the n-terln approxilllation 
fronl such bases or fralnes cannot surpa the rate of the best t-terlll Çourant (or 
box spline) approximation, but they may give better coustants and hence better 
performance results in pract.ical situations. 
It is also an important, observation that. even in the case of uniform triangula- 
tions, the B-spaces used here are different from the Besov spaces used in nonlinear 
approximation. For a nlore complete discussion of this issue, see [11] and [3]. 
Finally, we remark that in a related paper [12] we extend the argunlents of 
this paper to develop a corresponding approach in the Hausdorff nletric which is 
natural for approximating surfaces. There we also consider various practical aspects 
for decompositions, nmnerical approximation, and data structures. 



2626 B. KARAIVANOV. P. PETRUSHEV, AND R. C. SHARPLEY 

APPENDIX. COLORING LEMMA 
In order to kecp focus on the main analvtical results of the paper, we bave 
postponed the proof of thc coloriug lcnmia used iii Section 3.2 to this appendix. This 
decolnpositiolI result was used to create a manageablc collection of tree structures 
fol" estimatiug both the error and the mmibcr of elenients used in our coustructed 
approxilimlit. Silice this is a gelieral purpose result which lnay prove useful in 
similar settiligs, we give its proof iii full in this appcndix. Fol" clarity we ha,ce 
broken down the proof into a scries of lelnnIas. Since the coloring is donc in several 
refinenient stages, itis helpful to think of thc colorilig as an ordered triple of 
primary, secondary, and shadc colors. ŒElie I)rinIary coloring b'ill sort the elements 
pel'iodically by l'csolution lcvel, tlic secondary colorilig will insure there is spatial 
color separatiolI, and the third colorilIg (shadilig) is a niore delicatc adjustmcnt to 
ilisure that trce structures are formcd. Wc i)egin by rcpeating thc statcment of the 
coh)ring lcninia for the reader's convcnience. 

Coloring Lemma [see Lemma 3.2]. For any LR-triangulation 7- of E, the 
set 6) := 0(7-) of all cells g«nerated bg T can be r«presented as a finite disjoint 
(0)u=l with A" = h (,oE0, «Io) (,oEç is the mimal valence and 
union of its subscts "  "  
M. is the maximal tuntber of childre of a triangle in T) such that each " bas 
a kree structure with .respect to the inclusion lation, i.e.. if 0'. 0"  0", then 
( O'F  ( O"F # O. or O' c 0", or 0" c 0'. 

To begin the proof, we show, without loss of generality, that for the purposes of 
cololing we nIav assmne that the niultiresolution triangulation provides sufficient 
lesolution with each rcfinenient step. \Ve argue below that after a certain fixed 
liUlifl)er of ilicreliieltS of the levcl there will be a guaranteed refilienient of each edge 
and trimigle, which bv hypothesis is controlled fionl above, i.e., tllliforllllV bounded 
valences and nlax litllnber of subtrianglcs for each refinenlent. Consequently. we 
may separate the levels of O into L (L := [12NO 1112 /0]) disjoint classes (prilnary 
colors) by placing two levcls iii tlie Saille class iff their indices are the saine (nlod L). 
Thus a class  is of the fornl ( 11 Uj=0 j, where (0 := Ojo for sonie 0 < j0 < L 
and (j := (Ojo+j L. Since each such class  has a different priniary color, it will 
suffice to show how to dcsigna.te thc secondary colors of the nielnbers of a single . 
Thercfore, to sinipli" the notation aud wording of argunicnts, we will silnply refer 
to (secondary) coloring tlie classes  ilistead of O. In Lenuna A.1 below we show 
however that tliese classes have additional useful pl'operties. Loosely speaking, 
part (a) shows that the old vertices on a given level are far apart iii terlns of the 
graph nietl'ic. Iii part (b) a similar statement is given for the "'central parts" of 
non-overlapping edges of Curant elelnents fi'oni different levels of (. 
For D C ]2 and m _> 0, we define the star Stm(D) inductively by çt°(D) := D 
and Stm(D) := U{0 G o • level o = rn, 0 ° f3 St-I(D) ¢ 0}. For the vertices in 
resolution level m. this is just the neighborhood of radius k in the graph metric. For 
an edge e with vertices v  and v" and au integer m > level e, we define the "central 
2 R--1 ,t 
part" oftheedgetobest(e,m):=Stm(e\Str ({t ,v"})),whereR:=,., o +4. 
This selection for R has been lnade sufficielitly large so tha.t part (b) of the following 
leunna holds. 



NONLINEAR PIECEVt'ISE POLYNOMIAL APPROXIMATION 2627 

Lelnlna A.1. The Courant collection ) d«scribed above sati.iïes the follou,ing con- 
ditions: 
(a) For ea«h edge [v, v'] the distance between v and v'. rneasured in the graph 
metric on the next finer lcvel of (--). is af least 4R. 
(b) If e and e  are edges frein cells in ). m is an i'nteger with m - L >_ 
level e _> lcvcl e', and e  e'. then st(e, m) ç e' = (3. 
Pro@ (a) Note that each cdge in (-) gcts subdivided at lcast once after 2N0 lcvels. 
Further, observe that al'ter / := 2\ refincmeltS of any triangle, uone of its 
vertices can t)c COlme('tcd te thcir oI)posite edge bv a single edge at the finer level. 
Using this observation repcatcdly, one ca.n verify that after L refinements, the graph 
nletric distance I)etween t, and v  will I)e at least 2 L/l° = ]lI3 Nô > 4R. 
(b) Let v and v  I)c the vertices of e. Using twice the observation frein the 

proof of part (a), we conchlde that the distance frein each of the vertices in 
e \ ,çtl_2No({V,v'}) to e' is at least 4 when lneasll,ed iii the graI)h nletriç on 

level m. Therefore. on the m-th level, e \ stl_l%({v,v'}) has a. buffe, of at least 
three layers of triaugles lhat separates it fl'Olll Ct. O11 the other haud. the existence 
of ]il0 and th« choie' of R gUa.lalit'e that 5t, ({v. v'}) D St_2N({v.v'}), and 
this estal)lishes the «la.iln.  

This COlnpletes the i)rimary colol'ing, and fi'om this point on we onlv need work 
with a particula.r (2) (i.e., a fixed primary color). In this case "level 0"" will now refer 
to the level of 0 in (2) rather than in (-). as will the star St(O) and st(e, m). Also. 
when referring to the color of a. ('ell we will now lnean the secondarv color, mfiess 
otherwise specified. For 0  ) we denote by 00 the I)oundarv of 0. and bv x0 the 
central point of 0. We say that the cells in -(=)' C (2),, are R-disjoint (R >_ 1) if 
0 ° ç St(O') = 0 for anv 0, O'  '. 
The next result is used for the (secondary) coloring of cells of ). proceeding from 
coarse to fille levels, and uses M colors, so that sanie color cells are R-disjoint. 
Lemma A.2. Suppose some of the cells on a 9iven level are colored in ]iI := 
jyô+l _]_ 1 (R >_ 1) colors se that the saine celer cells are R-disjoint. Then the test 
of the cells on that level can ha colored in the saine ]il colors se that the saine celer 
cells are R-disjoint. 
Pro@ Te complete the coloring Oll the given level, we first use celer #1 te paint 
as many cells as possible se that the saine celer cclls are R-disjoint. Next. we 
use celer #2 as lnuch as possible, followcd by the third and se on until either all 
cells get painted or we run out of colors. The latter case, however, ncver occurs. 
Indeed. assume the contrary and let. 0 be the first cell that cannot be colored by 
this algorithm with the ]il colors. The cell 0 has the property that xvitllin its R + 1 
star St+l(o) there lnust be at least one cell paiuted with each of the ,'il colors. 
But this contradicts the fact that ]il was selected te be at least as large as the 
number of cells within ç/Rm+l(0 ). [] 
For the secondary coloring we proceed inductively, begilming at the coarsest level 
0, and color cells in M colors so that sanie color cells are R-disjoint. Suppose 
then that all levels up t.o "-. (/," > 0) have been colored. We color . as follovs. 
Step a) (Color corner cells). First we define the notion of corner cell. A cell 0 
of level k is called a corner cell for a coarser cell 0  if O' has ail adjacent cell 0" (a.t 



2628 B. KARAIVANOV. R PETIRUSHEV, AND R. C. SHARPLEY 

FIGURE 2. ('orner oeils fl'Olll Step a) 

the smne level of course) so that a'0 lies on edge [Xo,. xo,,] and xo is adjacent to xo,, 
on the level /," (see Figure 2). Given a cell 0' E (2)k-1, we color each of its corner 
cells 0  )a. the saine color as 0'. This insures that a cell's color is propagated 
through all fincr levels to its corner cells. 
Step b) (Ex'tend the colorir.q fo R-stars of the vertices on level (k- 1)). For 
each vortex ' on level (k- 1), wc paint tho cells contained in çtR+2rv 
colors so that the coloring donc in Step a) is preserved and ea«h color is used at 
nlost once. This is alwavs possiile, since Al was selected sufficiently large. Note 
that after this ste 1) lhc saine «olor cells are R-disjoint, since part (a) of Lemnm A.1 
guarantees that the stars are sufficiently separated. 
Step c) ( Complete the seco¢tdar'9 colorm9 of 0.). Accounting for the cells 
previously Iminted in Steps a) and b), we color the remaining cells from . as 
described in Lenmm A.2. 
This procedure specifies the secondary coloring of (2). and we have thus repr 
sented it as a finite disjoint llnion Uv=l ), where are all cells (secondarily) 
colored in the -th color. Thus the prilnary color skips levels until sucient refine- 
lnent is guaranteed, while the secondarv color insures sucient spatial separation 
on each level to control cell overlaps. Unfortunately. the collection of saine primary- 
secondary cololed cells (") lnight not fornl a tree structure, i.e., there lnight be 
two cells in "whose inteliors lneet but neither of theln contains the other. This 
lnaV only happen when a fincr cell lies on tlle edge of a given cell. To fix this defect 
we will set for each fixed )" the third coloring component, the hade of the cells, 
from two possible choices. First. we say that 0' and 
edge of the finer of the oeils is contained iu an edge of the coarser. 
X now restrict our cells to be of fixed prilnary and secondary colors (i.e.. fix 
and inductively determine the shade of these cells. On the coarsest level  of " all 
cells are disjoint, and we assign them shade #1. For the induction step. we suppose 
cells of all levels of "up to level k have been shaded and each shaded collection 
satisfies the desired tree properties. We say that a cell 0 is shade-consistent with 
if xo does hot lie on an edge of any cell that has the saine shade as 0. Hence it is 
possible to place 0 in this shade collection and preserve the tree structure. In this 
case we will also use the terlninology that 0 is consistent with that particular shade. 
We now proceed to shade the cells belonging to level k, i.e., 0  , accordiug to: 
Case i. If 0 both touches and is shade-consistent with some coarsel" cell . then 
we assign to 0 the saine shade as that of the finest such 0. Recall that 
this finest cell is unique by the construction of . 



NONLINEA1R PIECEWISE POLYNOMIAL APP1ROXIMATION 2629 

Case ii. Othcrwise, we assign to 0 the fil'st lllnllbered shade for vdlich 0 is COll- 
sistent. If no such shade exists, we introduce a new shade for (. 
Bv the constructiol iii the induction step. it is obviols that cach shade sub- 
collection has the desired tree structure. \\'c will show that these criteria intro- 
duce at lnost two shades. For this we need a couple of technical facts. \\'e relnind 
the reader that all cells bclong to a fixed )', i.e., thev havc a fixed prilnary and 
secondarv color. 

Lemma A.3. If (4 iutersects au ed.qe c' of a coarser cçll O' btt is ,ot oue qf ils 
Pwof. Let e' be an edge of 0' that intersects 0. and let , be a vertex of c'. Bv 
Step b) of out coloring procedure (for secondaly colors), St(t,) contains a ceci'ner 
cell 0" iii ),n that is shaded the Saille as ' . BV Step c) in the COllStlllCtiOll of 
St(v) does hot contain any other cells ff'oin ). Since O is hot a cOrllt,r cdl of 
R-1 
then 0 # 0". Therefore #  SI,, (t,) = , and so O nmst lll('et gl 
where t,' is the relnaining vertcx of e'. Th'rcforc, O C st(e', ). 
Lemlna A.4. Çell of )" ,ith di>rettt atades do ,ot totch. 
Proqf. Suppose to the contrarv that cdls 0 a, 0.  " of different shades (shadc 
shade #k, respectively) do touch. > mav rira assume that Oj is a nmxilnal (i.e.. 
coarsest levcl) cell of shade #ff that touches 0t., and conversely, that 0t. is a nlaxilnal 
cell in shadc #k that touches Oj. This follows bv iteration and the fa.ci lhat there 
are onlv finitdy manv COal'ser levels; so the iteration lnUSt terlninate. 
 lnay assume without loss of generality that lcvel Oj < lexx,1 Or. =: ttt-, and 
let ej.ck denote the edges of 0j.0t. respectixwly, such that ek C ej.  consider 
the two cases under which the finer cell 0t. could bave been shaded, and show that 
each one leads to a contradiction. 
For Case i. In this event there would be a coarser cell   " of shade #k that 
touches 0k and to which 0t. would be sha.de-consistent. Le êk be an edge of 
where it is touched by 0k. > consider two possible subcases, depending upon the 
relative level of  to that of Oj. 
Subcase i.a k is filler than Oj. 
Since levd Oj 5 level t. < level 0, then by part (b) of Lelnlna A.1 either èk C e 
or st(êk, mk)  ej = . The first possibility nmy be ruled out, since it would ilnply 
that the coarser cell t. would touch 0j, but 0k is the lnaxilnal such cdl of shade 
Hence st(êk, mk) lnllSl be disjoint fronl ey. Note that Or. is 110[ a conler cell of . 
If that were the case, then 0k would be disjoint ffoin the interiors of all edges on 
level  except the edge on which «0 lies and the edges (at lllOS[ tWO, possibly one) 
where t- is touched by 0t.. Hence, ej lnUSt overlie one of the these edges, since it 
contains e. This, however, contradicts the fact that 0j touches 0k in the forlner 
case and contradicts the inaxilnality of 0k in the latter. Therefore 0k CallllOl be a 
corner cell of , and so, by Lemlna A.3. 0 c st(è«..,). But we have already 
proved that st(êk, mk)  ej = , which is ilnpossible, silce 0k touches 0 
Subcase i.b k is coarser than Oj. 
Since level  < level 0j < level 0k, then again by part (b) of Lelnma A.1 either 
ej C èt. or st(e, ttz)êk = - The former case contradicts maxilnalitv of 0t. relatix 
to 0 a. For the latter cse, note that 0k cannot be a corner cell of Oj, because 



2630 B. KARAIVANOV, P. PETRUSHEV, AND R. C. SHARPLEY 

and Oj have difforent shades. Therefore, by Lemma A.3.0,. C st(ej, mk), and so we 
obtain 0k  êk = , which is impossible, since 0k touches  on êk. 
For Case il. If lris case occurred for the shading of 0k, then since 0j is both 
coa.rser than and touches 0, 0 must hot have been shade #j consistent. Hence 
there lnUSt be a Oj  " of shade #j that is coarser lhan 0k, and «0 belongs to 
some edge ê of j. X consider two possible subcases, depending upon the level 
of j relative to that of 0. 
Subcase ii.a Oj is coarser than Oj. 
Since level 0a  level j < level 0», thon compare edges e, êj using part (b) 
of Lenmm A.1 to infer eilher st(ê,mt-)  ej =  or êj C ej. In lhe latter ce, 
it follows that both the edge ek (rocall Ok touches the coarser Oj on ek) and the 
opposite vertex 0 (since »  ê) of a triangle in  are contained in e, which 
is clear b" impossible. If the former case holds, i.e., st(ê, mk)  e = , then a 
contradiction also results. To see this, observe that 0 cannot be a corner cell for 
a, due to the fact tiret they have different shades. But Lelnma A.3 implies that 
O C at(êj, mb-), which contradicts the fa«t that 0  e # 0. 
Subcase ii.b Oj is finer than Oj. 
Since level j < level Oj < level 0t-, we again compare edges èj. ej /lsing part (b) 
of Lemnm A.1 to imply either .st(ej, mg)  èj =  or ej C êj. By quite similar 
algmnent.s t.o the previous sui)case we can in'ove that contradictions are reached. 
Specifically, the latter statenmnt implies that both the central vertex 0» and its 
opi)osite edge et. belong to the edge êj. On the other hand. the fact that 0k cannot 
be a corner cell %r Oj will imply that Or, C st(ej, m), which will show that 0 
belongs to the intersection st(ej, mk)  êj, and contradict the former statement 
al)ove. 
By out smnption that different shaded cells could touch, we are led in ail ces 
to contradictions, thereby completing our contrapositive proof. 
By combilfing the previous results with the next lenmm, it follows immediatel3 
that  can be colored with K := 2ML colors, and the proof of the coloring lemma 
will be complete. 
Lemma A.5. Af most two shades are reqaired. 
Proof. Suppose in Case ii of the shading step a.bove that a third shade were needed 
for some cell 0. Then its central point 0 ¢ e  e2 for sonle edges e of 0 and 
e2 of 02, where 0, 02 ¢ " are coarser than 0 and bave shade #1 and shade 
respectively. Now, if «0 were a vertex for e, then there would be a corner cell of 
0 in " adjacent to 0. which is clearlv impossible, since cells at the saine level are 
H-disjoint. The saine reoning a.pplies to e2. Therefore 0 callnOt be a vertex for 
either e or e2, and we conclude that eï  e ¢ . Hence, 0 and 02 touch, which 
contradicts Lemnm A.4. 

REFERENCES 

[1] C. Bennett and R. Sharpley, Interpolatzon of operators, Pure and Applied Mathematics Vol. 
129, Academic Press, Inc., Boston, MA. 1988. MR 89e:46001 
[2] J. Bergh and J. L6fstr6m, Interpolation spaces: An introduction, Grundlehren der Mathema- 
tischen Wissenschaften, No. 223. Springer-Verlag, Berlin-New York, 1976. MR 58:2349 
[3] O. Davydov and P. Petrushev. Nonlinear approximation from differentiable piecewise poly- 
nomials, 2002. preprint. 



NONLINEAR PIECEWISE POLYNOMIAL APPROXIMATION 2631 

[4] R. DeVore, 1. Daubechies, A. Cohen, and VV. Dahmen, Tree approximation and optimal 
encoding, Appl. Comput. Harmon. Anal. II (2001), 192-226. MR 2002g:42048 
[5] R.A. DeVore, B. Jawerth, and B. Lucier, Surface compression, Computer Aided Geometric 
Design 9 (1992), 219-239. IkIR 93i:65029 
[6] R.A. DeVore and G.G. Lorentz, Constructive Approxzmation, Grundlehren der Mathemati- 
schen Wissenschaften, Vol. 303, Springer-Verlag, Heidelherg, 1993. MR 95f:41001 
[7] R.A. DeVore, P. Petrushev, and X. Yu, Nonlinear wavelet approximation in the space C(Rd), 
Pro9ress in Approximation Theory (A. A. Gonchar, E. B. Saff, eds.), Springer-Verlag, New 
York, 1992, pp. 261-283. IkIR 94h:410ï0 
[8] R.A. DeVore and V. Popov, Interpolation of Besov spaces, Trans. Amer. Math. Soc. 
305(1988), 397-414. MR 89h:46044 
[9] R.A. DeVore and V. Popov, Interpolation spaces aud non-linear approximation, in Function 
Spaces and Applications, M. Cwikel, .I. Peetre, Y. Sagher, and H. Wallin (eds.), Springer 
Lecture Notes in Math. 1302. Springer-Verlag, Berlin, 1988, 191 205. MR 89d:41035 
[10] M.A. Duchaineau, M. VColinsky, D.E. Sigeti, M.C. Miller, C. Aldrich, and M.B. Mineev- 
Weinstein, ROAMing Terrain: Real-time Optimally Adapting Meshes. Proc. IEEE Visual- 
ization '97, October 199ï, pp. 81-88. 
[11] B. Karaivanov and P. Petrushev, Nonlinear piecewise polynonlial approximation beyond 
Besov spaces, 2001, preprint. (http://www.math.sc.edu/ïmip/01.html). 
[12] B. Karaivanov, P. Petrushev and R.C. Sharpley, Algorithms for nonliuear piecewise polyno- 
nfial approxitnation, 2002, pr(,print. 
[13] P. Petrushev, Direct and converse theorems for spline and rational approximation and Besov 
spaces, in Function Spaces and Applications, M. Cwikel et. al. (eds), Vol. 1302 of Lecture 
Notes in Mathematics, Spriuger, Berlin, 1988, pp. 363-377. MR 89d:41027 
[14] P. Petrushev, Multivariate n-terre rational and piecewise pol,vnomial approximation, J. Ap- 
prox. Theory (2003), to appear. (http://www.math.sc.edu/ïnfip/01.html) 
[15] P. Petrushev and V. Popov, Ratzonal approximation of real functions, Cambridge Universitv 
Press, 1987. MR 89i:41022 

DEPARTMENT OF IOETHEMATICS, IJNIVERSIT OF SOUTH C,ROLINA. COLUMBIA. SOUTH CAR- 
OLNA 29208 
E-mail address: karaivan@math, sc. edu 

DEPARTMENT OF ]IATHEMATICS, UNIVERSITY OF SOUTH CAROLINA. COLUMBIA. SOUTH CAR- 
OLINA 29208 
E-mazl address: pencho@math, sc.edu 

DEPARTMENT OF ]klATHEMATICS. UNIVERSITY OF SOUTH COEROLINA. COLt MBIA. SOUTH CttR - 
OLINA 29208 
E-mail address: sharpley@rnath, sc.edu 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETh 
\olume 355, Number 7, Pages 2633-26-19 
S 0002-9947(03)03271-9 
Article electronically published on February 27, 2003 

THE ALMOST-DISJOINTNESS NUMBER MAY HAVE 
COUNTABLE COFINALITY 

.IRG BRENDLE 

ABSTRACT. \Ve show that it is consistent for the ahnost-disjointness lltnllber 
a to have couiltable cofinality. For exainple, it nlay be equal to 

INTRODUCTION 
Cardinal invarialltS of the COlltilllllllll, that is, cardimd mlmbers lmtxveen RI and 
c (the size of the COlltilllllllll) which atle defined a.s the smallest size of a familv of 
real nmnbers with a certain comhinatorial prolmrty, play an increasingly ilnportant 
role in moderl set theory. Equalities and inequalities between cardinal invariants 
have many commotions with problems arising lmturally in general topology, real 
analvsis and algebra, and, ri'oin a purely set theoretic point of view, there is a 
deep interplay with forcing theory, in particular iii the light of the search for new 
iteration techniques. 
One of the most basic questions al»out cardinal invariants is which values thev 
can assulne, and, for almost ail cardilmls, it is klown that anv regular value is 
possible, t Furthermore, most cardinals tan either be showll to be regular iii ZFC 
or they are equal to c in the l'andom real model, in the Çohen real lnodel. 2 or even 
in both, so that they can be consistently singular of uncoultable cofinality. Notable 
exceptions are the splitting llUlnber  of which it is still ullkllown whether it mav be 
singular [V] and the almost-disjointlmss lmmber a. which has recentlv been shown 
to be consistently singular of uncoultable cofinalitv bv Shelah [$2].  Things get 
trickier when one considers singular cardinals of countable cofinalitv. In fact, bv 
far inost of the cardinals, even those singular in the Cohen o1 randoln lnodels, can 
be shown to have uncountable cofinality in ZFC. 4 Exceptions are 
• Shelah [S1] has proved that the covering lmlnber of the mdl ideal may have 
countable cofilmlity, 
•the ahnost -disjointness mmber a dealt with below. 

Received by the editors October 3, 2001. 
2000 Mathematics Subject Classzfication. Prinlary 03E17; Secondary 03E35. 
Key words and phrases. Maximal almost-disjoint familles, almost-disjointness number, iterated 
forcing. 
Supported by Grant in-Aid for Scientific Research (C)(2)12640124, Japan Society for the 
Promotion of Science. 
lin fact, most of the cardinals that have been studied are equal to « under Martin's axiom 
MA. 
2The models obtained by adding  = + many random or Cohen reals 
3Cardinals relevant for this paper will be defined below. Also see [vD], [V], or [BI]. 
4This is also true for . 

@2003 American Mathematical Society 

2633 



2634 .I)RG BENDLE 

• itis unknown whetller the reaping mmlber r [M. Problem 3.4] or the inde- 
pendence number i can have countable cofinality. 5 
Here we show 
Main Theorem..4.sume CH nd let Abe  singlr crdil of coq.table cofi- 
,ality. Then there is a forczg extension satisfyig a = A. In particular, a = R is 
consistent. 
 give a brief outline of the proof. Itis well known that, assuming C. one 
Call force a mad family of size R such that, by a standard isomorphism of names 
argument, there is no mad familv of size R for n k 2 in the generic extension [H1]. 
However, mad familles of size R may survive the forcing. The sinlplest wav to 
get rid of small mad familles is by iteratively adding dominating reals, sav for 
R2 steps. If this is donc in such a wav that every dominating real is dominating 
only over a fragment of the mad family of size Rw a¢l¢led in the initial step. the 
latter will survive. %t, if the dominating reals are added in a standard way. 
the t})rcing will lose most of its homogeneity, thc isomorphism of-names argument 
will ceae to work, and there may he a mad fimfily of size R2 instead. This is 
where Shelah's rcccnt technique of it«,ration along templates [$2] cornes in. 6 It 
provides for a. way of adjoining dominating reals with a forcing having enough local 
homogencity. However. since isomorphism-of names arguments require CH iii the 
ground model, we need to describe the two step extension sketched above in one 
step and incorporate the forcing adding the mad familv of size R into the template 
framework. This accounts for some of the tcchnical difficulties described below. 
Apart from proving the Main Theorem, we also present a new, more axiomatic. 
treatment of the teml)late framework (Section 1). While Shelah [$2] defines the 
teml)late via two procedures building more complicated sets (with larger depth) 
from simpler ones. we onh" require that the template is a family of subsets of 
the linear order underlying the iteration satisfying several axioms, most notably 
well-foundcdness.  think this apl)roach is more lucid, apart from providing for a 
simpler definition of the iteration. There is a price one bas to pay for that. however: 
the Completeness Lemma 1.1. showing that we are indeed dealing with an iteration, 
requires some additional work. Of course, our general approach can also be used 
to prove Shelah's original results [$2] (sec [Br]) and. siuce thcre is no need to add 
a mad family in this case, definit.ions will l»e simpler than in Section 1. 
In Section 2 we describe the tcmplate used for the proof of the Main Theo- 
rem, and in Section 3 we provide the isomorphism of nalnes argument needed to 
comp}ete the proof. 
The template framework developed in Section 1 for Hechler forcing can in fact be 
used to handle a large class of easily definable ccc forcing notions (see [Br, Section 
4] for a few examples). eplacing the forcing generically adjoining a mad fanfily 
bv ail appropriate relative, one can get analogous results for relatives of a. e.g., 
for as, the size of the smallest mad familv of partial flmctions from w to w. So, 
for example, as = Rw is consistent, and so is a = R < as = Rw. For the latter 

5As for maximal almost-disjoint families, it is rather easy to force a maximal independent 
family of size, say, R. Both r and i are equal to ¢. and thus possibly singular, in the Chen and 
random models. 
6As mentioned above, Shelah showed among other things that « could be singular. In his 
models, « is equal to ¢ and therefore has uncountable cofinality. 



THE ALIvIOST-DIS.IOINTNESS NUIvIBER 2635 

result, one needs to replace He«hler foreing by eventually-differeut-reals forcing in 
the framework of Section 1 (cf. [Br, Sectiou 4]). 
Let us briefly recall the maill notious relevant for this paper. Two infinite subsets 
A and B of w are called almost-disjoint if their iuterse«tiou is filfite. A ç [w]  is 
an almost-disjoin, t family if its lnemlmrs arc pairwise almost disjoint. A is a mad 
family (maximal almost-disjoint family) if it is lnaximal with respect to beiug an 
alluost-disjoint faufily, i.e., for everv B  [w]  there is A  A such that A  B 
is infinite. The almost-disjointness number a is the size of the least infinite mad 
fmnilv. For functious f,g  w , we say that g evedually dominates f (and write 
f * g) if the set {n; f(n) > g(n)} is finite. The unbounding nu,nber b is the 
cardinality of the smallcst unbounded falnily in thc structure (w , *), that is, the 
size of the smallest U ç w such that for all g G w  there is f G U with f * 9- 
The dominatin9 number 0 is the size of the least cofinal fitmily in (w , *). It is 
well known and easv to see that b  0 aml b  a iu ZFC [vD]. 
He«hler for«i,,9 [H2] D (see also [B.I]) «onsists of pairs (s. f) where s G w <, 
f G w w and s ç f, ordered 1,y (t,g)  (s, f) if t  s, and g k f everywhere. 
It generically adds a domin.ati 9 real, that is, a real that eventuallv dominates all 
ground model reals. It is this forcing adjoining a dominating real which we shall 
use in the template fi'aluework sket«hed al»ove. 
Oto" notation is standard. For cardinal invariants of the continuum, we refcr 
to [vD], [V] or [B1]. For forcing theory, in particular for forcing related to cardinal 
invariants of the contimmm, see [BJ]. 
I thank .huis Steprgns fi»r pointing out a flag iu an earlier version of this work. 

1. TEMPLATES AND ITERATIONS 

The lnOSt useflfi definition of a template seems to be (see also [Br]) the following. 

Definition (Template). A template is a pair (L,Z) such that (L, _<) is a linear 
order and 2- C_ 7)(L) is a family of subsets of L satisfying 
(1) {3, L e Z, 
(2) 2- is closed under finite unions and intersections, 
(3) if y < x beloug to L, then there is A G 2- n 7(L) such that y G A. 
(4) ifAG2-andxGL\A, thenAnLG2-, 
(5) 2- is well-founded, i.e., there is a fuuction Dp = Dp/" 2- -- On, called depth. 
recursively defincd by Dp(0) = 0 and Dp(A) = sup{Dp(B) + 1: B G 2- and 
/3cA} forA•2-\{{3}. 
Here, Lx = {y • L; y < x} is the restriction of L to x. IrA C_ L, we define 
ZIA = {B ç A: B • Z}, the trace of 2- on A. (A, ZIA ) is also a template. 

Since 2- is closed uuder finite intersectious, if A • Z, theu ZIA = {B • Z: B C 
A}. 
In our context, we need to slightly revise this definition because we will hot 
only have "iteration coordiuates" LHech used for adjoining Hechler generics but 
also "product coordinates" Lmad used for adding a mad family. Siuce the former 
should be generic over some, but hot all, of the latter, we need to incorporate them 
into the saine telnplate fi'alnework L, and some of the above clauses should be true 
for all of L, while others need to be satisfied only for members of LHech- 



2636 JIRG BIRENDLE 

Accordingly, let LHech and Lmad be disjoint sets, put L = Lnech U Lmad, and 
assume L is equipped with a. linear order. Further suppose 27 _C (L) satisfies, in 
addition to (1) and (2) above, the followillg clauses: 
(3') if .r G LHech and g G Lx, then there is A G Z  P(L) Slch that g G A; 
(4') if A 6 Z and .r 6 LHech  .4, then 4  L 6 Z; 
(5 ) the trace Z[LHech = {A  LHech; A G Z} is well-founded. 
as well as 
(6) if A G Z and .r G A, then L  Lma d ç A. 
This is the definition of "telnlflate" we shall work with for the relnainder of the 
imper. Notice that (5 ) lllea.llS in tmlticular that the depth functioli Dp depends 
only on the LHech-part, i.e., Dp(A) = 0 iff A ç Lma d and, recursively. Dp(A) = 
sup{Dl)(B) + 1: B G Z and B  LHec h C A  LHech}. 
(6) is a closure condition for the L,,,aa-parl which is needed to make the proof 
of glain Lelmna 1.1 l»clow go thlough, glore gclel'ally, we say that A ç L is «losed 
if A satisfies (6). (Se» Z consists «mly «»f cl»sed sets.) r arbitrary 4 Ç L, we then 
(Icfin« it.s «lo.ure c'(A) by c,(A) = A U .en (L  Lmad)- Thus c'(A) is the slnallest 
closed sel conla.ining A and al(ci(A)) = d(.4). 
The basic idca fiw lhc following, attelnpted, dcfiniti«m con,es from [$2]. It is 
modifie(l, however, due te our axiolnatic treatment of the concept of "template'" 
(see also [Br]) and because of thc inclusion of Lm. 
Definition (Iterating Hcchlcr forcing and adding a lnad falnilv along a telnplate). 
Assulne (L,Z) is as above. X> define, for 4  Z, bv recursion on Dp(A). the 
partial ordcr (p.o.) P[A (more explicitly, we define P[(A. Z). but we shall drop the 
reference to Z in case there is no ambiguity). 
• Dp(A) = 0. This mcans A ç Lm- P[A consists of all finite partial 
fulctions p with domain containcd in 4 and such that p(z)  2  for all 
:  doln(p) for Solne u = tv  w. The ordering on P[A is given by: 
q et.4 p if doln(q)  doln(p) a.nd 
- ,tq  , p(z) ç q(z) for all z  dom(p), I{z  doln(p); q(z)(i) = 1}1  
1 for all i  n q . 
• Dp(A) > 0. PI4 consists of all finite partial flmctions p with domain 
contained in A and such that 
- therc is t = tv  w with p(z)  2  for all z  doln(p)  Lmad; 
- letting .r = lnaX(dom(p) Lh). there is B  Z  P(A  L) (so 
 sP 
Dp(B) < Dp(A)) su«h that p[(A  L)  P[B and p(«) (,f) 
where s G w < and f is a P[B-name for an elelnent of w, such that 
p(A  ) ,eu ". ç " (thi mlS (.) i a et-m r for a 
condition in Hechler forcing ). 
The ordcring on P[A is given by: q etn P if dom(q)  doln(p) and 
- nq  n , p(z) ç q(z) for all z  doln(p) Lm. {z  dom(p) 
Lm; q(z)(i) = 1}  1 for all i G nq  " (this guarantees that the 
reals added in COoldinates froln Lma d are characteristic fulmtions of 
an allnost-disjoint falnily),  well  

7Note. however, that the first coordinate sPx of the condition p(x) is not a name. 



THE ALMOST-DISJOINTNESS NUMBER 2637 

- either y = max(dom(q) ç LHech) > d:, - max(dom(p) ç LHech ) and there 
is B E Z ç) P(A  L) such that p[(A  L),q(A  L)  P[B and 
q[(A  Lu) IB p[(A  L), 
- or x = max(dom(q)  LHech) = max(dom(p)  LH¢ch) and there is 
B  Z(A  L) such that p[(A  L),q[(A 
L) Net pi(,4  Læ), ],f are [B nalnes, 
L) Iel "](n)  y(n) for all ,'" (the last two clauses mean that 
q[(A  L)Iet "q(x)  p(')"). 
 bave hot argued yet that this recursive definition works at all. Thc point 
is this requires that all [A's be transitive, which is hot trivial because the sers 
B G Z witlmssing that q  p lnav dcpcnd on the pair (p, q). Therefore, to prove 
transitivity, we need to show that the [B COlnpletely embed one into the other. 
This will be donc in Main Lelnlna 1.1 below. 
Note that, once this is achieved, [(A,Z) = [(A. Z[A) is imlncdiate for A 
Of course, the above re«ursion also defilleS [A = [(A,Z[A) for arbitrary A ç L. 
If A. B G Z, A C B, thon [A C [B is immcdiatc from thc definition (becausc 
Z[A C Z[B in this case). This is lnuch lcss «lear if one of A or B does hot »elong 
to Z. Neither is it clear whether [A <o [B, the lnost basiç prol)erty thc al»ove 
recursive dcfilition must satisfy to nmke it an iteration, ex-en in case ln»th A and 
B corne from Z. This issue is addressed bv the following crucial lelmna. 
Main Lenmm 1.1 (Colnpleteness of elnbeddilgs). Let B  Z and A C B be 
closed. Then [B is a partial order, [A C [B and even [4 <o [B. More 
explicitly, any p  [B bas a cammical redu«tion P0 = Po(P, A. B)  [A such that 
(i) dom(p0) = dom(p)  A 
(il) s ° = s for all x G doIn(p0)LHech and po(x) = p(x) for all x G doln(p0) 
Lmd (in particular, po = p ), 
and such that, whenever D  Z. B,Cç D, C close& C eends Po, then there is q G [D exte'nding both qo a,d p. 
Note that we do not require p B P0- 
Pro@ By recursion induction on , silnultaneouslv for a.ll templates (L,Z), 
• we prove that [B is indeed a p.o. (i.e., transitivity holds) for all B 
with Dp(B) = a; 
• we prove [A C [B for all B G Z with Dp(B) = a and all closed A C B: 
• we construct P0 = Po(p,A, B) satis[ving (i) and (il) for ail B G Z with 
Dp(B) = , all closed A C B and all p  [B: 
• we prove that for all B,D  Z, B,C Ç D, C closed, with Dp(D) = 
and all p G [B, letting A = C B and P0 = po(P, A. B) (which has been 
constructed either at stage a or at an cmlier stage), there is q  [D as 
required. 
Notice that if Dp(B) = a, D = B and C = A. then the latter indeed shows 
[A <o [B. 
The case a = 0 is trivial. So assulne  > 0 and Dp(B) = a.  first check 
transitivity of [B. Assume that r etB q etB P- Then clearly doln(r)  doIn(p), 
n r  n p, p(z) ç r(z) for ail z G doln(p)Lmaa, and I{z G doln(p)Lmaa; r(z)(i) = 
1}l  1 for all i G n r  n p. Let z = nmx(dom(r)  L«ch), y = max(dom(q) 
LHech) and x = max(dom(p) LHh). Then there are A0, A G Z  P(B  L,) 



2638 JRG BRENDLE 

such that pI(B ç Lz),q[(B ç Lz) • P[Ao, qI(B c3 L) -<et,% pI(B G L), and 
is a P[Al-name and. in case 9 = z, ¢g is both a [A-nalne and a PA0-name 
as well as, in case z = 9 = z,  is a PA0-name. Let 4 = 40 U Al. Then 
A Ç ZGç(BL) so that Dp(A) < Dp(B), and we know bv the induction hypothesis 
that PAi <o P[A for i = 0.1. Therefore, p(BGL),q[(BGL),r(BGL) 
and r[(B  L) PrA q[(B  L) PrA p[(B  L). Moreover,  is a P[A nalue 
and, in case y = z, ¢g is a PrA llalnP and r[(  L)IbtA fg   as well , in 
ce z = 9 = z,  is a Pt.4 llalllC glld q[(B  )Ikr f  g. Taken together. 
this shows that r tB P as requiled. 
Now let A C B  Z, A closed, bc given. Assmnc r  PA. Let z = lnax(doln(r)G 
LHech)- Bv definition of the iteration thcl'e is A  (Z[A)  (Lz) such that r[(A 
L)  Pli and  is a P[A naine. There is   Z[B ç Z such that 
Clcarly z  . Bv clause (4') in the dcfinition of a tcmplate, we may therefore 
assume without loss of generality that B ç Lx. Thus  C B and Dp() < 
Dp(B) = a. By induction hypothesis, PI,4 C P and [ <o P[. Therefore, 
'Ç is a P[ naine as well. r G P[B follows imluediately. Hence PI 4 C 
required. 
Next assume also p  PIN is given.  construct P0 = Po(p.-4, N). Put z = 
max(doln(p)  LHech)- By definition of the iteration there is   Z  ç(B  L) 
suchthat=p[(BL) F[and jisaF naine. Put A= A. Note 
that   Z[A. Bv indu«tiol hypothesis, ç has a reduction 0 = P0(ç, -, 
sat.is-ing the barred version of the clauses of the lnain lelnlna. The definition of 
p0 split.s into t.wo cases. 
Case 1. z  4. Then dom(p0) = doln(p)  .4. p0[(A  L) = 0 and po(z) = p(z) 
for z Ç (dom(p)  A)  Lx. (Note that such z must belong to Lmad-) 
Case 2. z e A. Then let doln(p0) = doln(p) A. P0 t(-4  L) = 0 and P0 (z) = p(z) 
for z  (dom(p)  A  Lmad)  L x.  know by induction hypothesis that PA 
<o [. Therefore. there exists a caomcal projectiott o to . of the F- 
naine . Accordingly we let p0(c) = ,,(P 
More explicitly, we do the following. For silnplicity work with the cBa's A = 
r.o.([.) and  = r.o.([) associated with . and .  know by the induction 
hypothesis that A <o . Note that   s ç . In , for all 
with s ç s, we let b s ç . So b = pand. for n > ] , the b, 
Isl = n, are a maximal antichain below . Let a be the product (intersection) of 
0 and the projectio of b t.o A (recall the projection of bs to A is the unique 
condit.ion a such that a OE b and any extension of a in  is colnpatible with b). I11 
particular, a = 0 and {a; I1 = ) = 0 for  > I1. Defin  by recursion 
on n = ]s] as follows, a = a = 0 and, for n > I1, st,  = ¢_. 
j<s(-l) a* 
t(_)-(j)) (which is equal to at(_ ) • (a  j<(_) at(_F(j)) ). 
Then one can show by induction on  > Isl that the a, Isl = ,, are a lnaxilual 
antichain below P0- Therefore they canonically defiue a F-name o (that is, 
 s ç " 
' = {a, s' < s evervwhere, 
The main property of this naine is t.hat for ail s, a ; _ 
 ç , I'1 = I1) is a reduction (not necessarily the projection) of bs = 
 everywhr«  ç 1'1 = I1}. (This is so because (a)' = {«-a* s' 



THE ALMOST-DISJOINTNESS NUMBER 2639 

everywhere, sP C_ s, 18'1 = 181} is the prod,,ct of i#0 and t.he projection of b' s alld, by 
, '  (aS)' is trivial.) 
the definition of as a s 
This completes t.he defilfition of P0. Clauses (i) and (ii) are triviallv satisfied in 
each of the two cases. 
Now aSSulne B. D  Z, B. Cç D, Cclosed, are such that Dp(D) = 
p  P[B and P0 = P0(P, A, B). Let q0 tc P0, qo  PIC.  need to constru«t q. 
X = lnax(doln(p)  LHech), , A,  and P0 are as in the previous construction. 
Case 1. z  A. So x  C. Let g = lnax{z < x; z G dom(q0) 
can find Ê e Z[C  P(Lu) such that qo[(C  Lv) e F[Ê and  is a F[Ê naine. 
Thereis   Z[D ç Zsuch that Ê = C. Since.q  Ê and.q 
follows. By clause (4 ) in the (tefinition of a telnplate, without loss of generality, 
 ç L u. By clause (3') ill the defilfition of a telnplate filld  G (Z[D)(Lx) 
contailfing g. Let  =  U  U B ç L and ç = (  C) U Ê u , Ç L» By 
clause (2) in the defilfition of a template,  G Z[D ç Z and  G Z[C. Since 
Dpzrc(Ê)  Dt)z)c(Ç)  Dpz(D) < o, O,) = qo [(C  Lz)  P[OE 1)v the inductiol 
hypothesis. 0o O Po and Ç = . are ilmnediate. By the inductive assmnption 
for the barred version, there is q  P[ extending both O0 and . 
 define q such that 
• dom(q) = doln(q) U doln(p) U dom(q0), n q = 
• q(9) = q0(9) for all 9 ¢ (dom(q0) N Lz)  Lnch, 
• q(x) = p(x), 
• qo(z) ç q(z), q(z)(i) =0, for z  (dom(q0)  L)L.ma and i  nq n q°, 
• p(z) ç q(z), q(z)(i) = 0. for z  (dom(p)  (Lz U clora(q0))) Lma and 
It is straightforward to check Oint q  [D aud q ev q0- So let us al'gue that 
q ev p as well. Clearly n q  n p.  need to show that p(z) ç q(z) for ail 
z  dom(p)  Lma. This is obvious for z < z be«ause ç e . It is immediate 
by definition for z > x belonging to dom(p)  dom(q0). So assume z > x, z 
dom(p)  clora(q0). Then p(z) = po(z) ç qo(z) ç q(z), as required. Next fix 
i  n q  P. We need to check that there is at most one z  dom(p)  Lma with 
q(z)(i) = 1. By way of contradiction assume this is true for two distinct z0 < z. 
By construction we nmst have i  n q°, x < Zl and Zl  dom(q0)  dom(p). Hen«e 
Zl  A. Therefore z0 nmst belong to A as well because A is closed. Thus both z0 
final Z 1 belong to clora(p0). Tlfis means that q(z)(i) = qo(z)() 
whi«h «ontradicts q0 etc P0, and we are done. 
Case 2. x  A. So •  C. Find   Z[CP(L) such that ç0 = qo[(CL) 
and o is a [-name. Without loss of generality, , ç C. C =  is immediate. 
There is   Z[D ç Z such that ç =   C. Sin«e x  , we get x  . Bv 
(4'), without loss of generality,  ç L. Ve may also qsume  ç . Since 
Dpz( ) < Dpz(D ) = a, we can fi'eely use the induction hypothesis when dealing 
with A, , C, and . In parti«ular, ç0 ee P0- 
Now note tn,t we have s = s  ç s  and 0 Iee s  ç f. Let m = I%1 . 
Since also 0 Iee "ëq  >_ ] (everywhere)", we see that q0 Iete 
' ff m < s eA P where we let p = 
Hence we get a := a qo = 
(the canonical reduction of ç0 to F[: note here that Dp(Ç)  Dp()  , so 



2640 .JRG BRENDLE 

that/ indeed has been defined already). However. bv construction, a <-'tA/30 is 
nothing but a reduction of b := b',o = []m G s° p G  to . So there is 
ri+  P[ such that + 5t ç and ri+ t b (so that, in particular, 
and ç+ Ikt  ¢tm  s). Let  = p0(ç+,A. B) be the canonical reduction of 
ri+ to P. Then  eb ç;" Therefi»re ç and 00 have a common extension 
ï in D[Ç. Bv inductive assmnption for the bm'red +-version. there is 
extending both + and 
XX define q such that 
0 + 
• dom(q) = dom(0 +) U dom(p) U dom(q0), v= n 
• q() = qo() for all 9  dom(q0)  LUe«h xvith 9 > x 
• s = s  and . is a P[ naine sud, that 0 + lket  = max{¢, f}, 
• q,,(z) Ç q(z), q(z)(i) = 0, for z e (clora(q0)  Lx)  Lmaa and i e ,,q  n q°, 
• p(z) Ç q(a), q(z)(i) = 0, for z e (dom(p)  (L U dom(qo))) Lmaa and 
i  q k P- 
To sec tiret q  [D. note that + ItD s  ç J: by construction. It is then 
sraightfi»rward  check that q tD qu.P- In fa«t, for q ?tD P we argue  in 
Case 1 al,ove. 
Note that. as an immediate consequence of lain Lemma 1.1, we get that for 
arbitrary closed A ç B ç L, F[(A. Z[A) completely embeds into F[(B. Z[B). 
Lemma 1.2 (Chain condition). Let .4  . Ang mcomdable K Ç F[A bas an 
un«oudable certered subsct. 
Pro@ Bv a standard A system argument, it suces to show that 
n p = t. sç = s for all x  dom(p)  dom(q)  Lunch, and p(x) = q(x) for all 
z  dom(p)  dom(q)  Lmad. then there is a common extension r with dom(r) = 
dom(p) U dom(q), n  = n  = n . 

if x  dom(p) C/LHech, 
if x C dom(q) çl LHech 

raid 
r(«) = { p(.r) if « C dom(p) ç/Lnd, 
q(x) if x  dom(q) ç/Lma d. 
We do this bv induction on Dp(A). 
The case Dp(A) = 0 is trivial. So assume DI)(A) > 0. First assmne x 
mmx(dom(p)  Lch) < y = max(dom(q)  LHch). Then there is B  Z P(A  L) 
such that pL, q[L u  PB. and  is a P[B naine. By the induction hypothesis, 
ve get the required fiL u %et pL.q[L. Let r(y) = q(y) and let r(z) = p(:) for 
z e dom(p)Lmad, z > y. and r(z) = q(z) for z e (dom(q)  dom(p)) Lmad, 
Next assmne x = max(dom(p)  LHech) = max(dom(q)  LHch). Again there is 
B e Z  P(A  Lx) such that p ILs. q tLx e P tB, and ]. ] are P tB-names. Again 
we get rtL«. Let s; = s = s and ]" be such that r[L It ]" = max{j,]}. 
Also let r(z) = p(z) for z e dom(p) Lmd, z > g. and r(z) = q(z) for z 
(doln(q) k doln(p))  Lmad, Z > g. 
Lemma 1.3 (Embedding Hechler forcing). Let x  LH«h and A 
T5en t5e two-step iteration FA .  that canonically adds a Hec5ler-generic 
coordinate 3" over te generic extension via F[A contpletely embeds into FL. 



THE ALMOST-DISJOINTNESS NUMBER 2641 

Pro@ Let B = cl(A U {x}). I?[B embeds into P[L bv Main Lemlna 1.1. So it 
suftïces to show P[A * b <o F[B. This does hot follow from (file statelnent of) 
Lenmla 1.1 because A U {x} need hot be closed, but it is relatively straightforward 
from the proof of 1.1. 
More explicitly, given p  [B, there is   Z[B  (L) such that fi = pILe 
P[ and ] is a P[ llalllç. Without loss of generality, ,4 ç . By 1.1, P[A 
o P[B. Therefol-e, p h a CmlOlfical l'eduction ç0  [A. As in Case 2 of file proof 
of 1.1. there is a canonical projection ]Ço to P[A ,f ]. Define l'o e L4 *  by 
po[A = P0 and p0(z) = (s, ]o). As in Case 2 of the proof of 1.1. argue that any 
q0  P[A * b extemting ivo is compatible with p. 
This may badly rail in case A  Z because then [A * b lmed hot elnbed into 
Lemma 1.4 (Names fi)r reats). Assume iv  [L avd ¢ es a [L ,ame for a real. 
Then there is A Ç L tout, table such lhal. Icltin 9 B = ci(1), iv  [B, ad ] is a 
P[ B vaine. 
Pro@ The proof proceeds bv sinmltaneous inductiol on Dp(L).  ithout loss of 
gelmrality. Dp(L) > 0. 
Assume first p  [L. Let x = max(dom(p)  LHech)- There is C  Z  
such that p[Lx  [C and ] is a [C-lmme. Bv ilduction hypothesis, there is 
A0 ç C COUltable such that pile  [c1(,4) and ]ç is a. [cl(A0) lmme. Then 
p  [B where B = ci(A), A = Ao U dom(p). 
Assume llOW that ] is a [L naine. By cCC-lmSS (Lemma 1.2), there are 
i, n  } Ç [L and {kn,i  : i. n  } such that 
• {p,i; i  } is a nlaximal alltichail in [L for all n  w. 
By the previous paragraph, we can find countal)le sers A,i such that p,i 
[cl(A,.i). Put A = i. 4,.i, B = ci(A). Since [B <o [L (Lelmna 1.1). 
we can construe ] as a [B-nalne. 

Assume (L.Z) and (L. `7) are telnplates and Z C_ `7. \Ve sav that Z is cofinal in 
`7 if for all x 6 LHech and all _4 6 `7 A/)(L:) there is t3  Zç7a(Lx) containing ,4. 
The following is. iii a sense, a trivialitv. 

Lemma 1.5 (Cofinal subtenlplates). If Z is «ofirml in . then P[(L,Z) is forcing 
equivalent to P['(L,,7). 

Pro@ By induction on Dp(L) (in the sense of 2-), we argue that conditions in 
PI(L,:/) and conditions in PI(L,) can be canonicalh identified so as to vield 
forcing equivalence. Without loss of generality, Dp(L) > O. 
Let p  PI(L,). Put x = max(doln(p) ç? LHech). There is A   C P(L,) 
such that p = p[Lx ¢ P[(A.,7) alld ]P is a P[(A. ,,ï)-llallle. Since 7 is cofinal iii 
,7, there is B  2- ç/)(L) such that A _C B. Bv Main Lemlna 1.1. we know that 
PI(A, `7) <o ]PI(B, ,7) and, by the induction hypothesis, PI(B, .7) and P[(B.Z) are 
foI'cing equivalent. Therefore, we lllay construe /3 as a condition in P['(B,Z) and 
ff as a PI(B, Z) naine. Thus iv ¢ P[(L. Z). It is straightforward to verifv that this 
identification induces forcing eqnivalence. [] 



2642 Jt:IG Bt:IENDLE 

Proposition 1.6 (Adjoining a scale). Assume tt is regular uncountable, p C_ LHech 
is cofinal izt L, and La  Z for all c < p. Then I?[L forces b =  = p (i.e., there 
is a p scale). 

Pro@ For each a < t I, let J: be the lmme for the Hechler-generic adjoined in 
coordinate a of the iteration (see Lemma 1.3). By construction, the fa are forced 
te be wellordered by <_*. Let ) be a I?IL-name for a reM. Bv Lemma 1.4, there 
is A C_ L counta.lle such that ) is a I?[cl(A)-name. Since t is regular uncountable 
and cofinal in L, there is a < tl such that ci(A) _C La. Since La  27, fa is forced te 
dominate the reals i the generic extension via II[La and, a fortiori, it will dominate 
.¢ [] 

Proposition 1.7 (Adjoining a mad family). Assume L bas uncounlable cofinalitl 
and Lmad i8 c&nal it L. Then FIL cano,tically adjoins a mad family of size [Lmadl- 
Pro@ Let G be F[L-gcncric over the ground model. For x G Lmad define t = 
{tf e ; p(x)(t,) = 1 for seine p e G}. Let M = {};" .r e Lmad}- By definition of 
the p.o., M is an ahnost-disjoint fanfily. We need te check maximality. Se let ) be 
a FIL naine for an infinite mil)set of w and assmne bv way of contradiction that 
p forces that 2 is ahnost disjoint frein ail f. By Lemma 1.4 there is a countable 
set A such that p  F[d(A) and 2 is a F[d(A)-name. Since L has mwountable 
cofinality and Lmd is cofinal in L, thcre is " G Lmad such that al(A) Ç L» By 
Main Lemma 1.I we know that P[cl(A) <o P[L <o P[L. 
Find k0 and P0 etz P such that 
Put P0 = po[L. Clearly any , 9  dom(0)  Lmd, is a PL-name. Se we tan 
find k k k0 and Pi tz 0 such that 
for all .q  dom(ç0)  Lmd- Since 2 is forced te be infinite, we cau find 
and i0 k k such that p Ietz, i0  2. Without loss of generalitv, n  > i0. Then 
we nmst necessarily have (9)(io) = 0 for ail .q  dom(p0)  Lmd. 
Define a condition p by 
* dom(p) = dom(ç) U dom(p0), 
* p(z) = po(z) for ail z 6 LH«h 
* p(z) D p0(z), p(z)(i) = 0 for all i with 
Lmad  dom(p0), z > , 
{ 1 ifi=i0, 
p(z)(i) = 0 for all i 
It is straightforward te verify that p  P[L and that p Nelz P0. Since 

we have a contradiction. [] 



THE ALMOST-DISJOINTNESS NUMBER 2643 

. BU1LDING A TEMPLATE FOR ADJOIN1NG A MAD FAMILY 
For simplicity asstlme CH for the relnainder of the pat)er. 
Assulne A0 k R is regular, and A > A0 is a singular cardinal of countable 
cofinality, say A = Un An, the Anbeing regular, equal to A n , and strictly increasing. 
Also sui)pose no < A for n < A. As usual, It* denotes (a disjoint copy of) 
with the reverse ordering. Elelnents of i t will be called positive, and lnembers of 
are e9ative. For each n choose a 1)al'titiol A, = Ua<w, s,î such that each S is 
co-initial in ,. Also aSSUlne S,  A = S,c for m < n. 
The following definition is lnotivated by Shelah's work [S2]. 
Definition (Template for adjoining a mari falnily). Define L = L(A) as follows. 
Elements of L m'e lOnelnpty finite sequen«es .r (i.e., dom(z) Ç w) such that 
• z(O)  
• x(n) Ç ; U , for 0 < ,t < Ixl- 1, and 
• in case Ixl  2, if .r(Ixl- 2) is positive, then x(Ixl- 1) Ç AlœeI_ 1 u A, and if 
.411 - ) i negative, then .r(I.r I - l) Ç A* U 
Sgy .F e LHech if Ixl =  or (11 - ) e * 
Ixl-1 U lxl-l" Otherwise .r Ç Lmad- (This 
lneans that x e Lmd iff Ixl 2 oe a.d either x(x-2) is 1,ositive and x(Ix- 1) 
or .r([x- 2) is negative and '(Ixl- 1) < * 
- Ixl-1-) Equip L with the following 
lexicographic-like ordering: .r < y iff 
• either .rC y and (Ixl) i positive, 
• or  c . lld x(lyl)is negative, 
• or. letting ,, := nfin{m • .r(m)  y(m)}, either .r(n) ia negative and y(n) is 
positive, or both are positive and x(n) < y(n), or both are negative and 
.r(,) <a. y(n) (i.e., there are a < fl <  such that x(**) = ff* <a. a* = 
()). 
It is immediate that this is indeed a linear ordering. 
We identify sequences of length one with their ranges so that 0 is a cofinal 
subset of L. Sav x Ç LHech is relevant if Ixl 2 3 is odd..r(n) is negative for 
odd n and positive for even n, x(xI- 1) < w, and whenever n < m are even 
such that x(n),a'(m) < 1, then there are fl < a such that .r(n- 1) Ç S_ 1 
and x(m- 1) e S_. For relevant x, oet & = [t(Ixl- 1),x), the interval of 
nodes between .ri(Ix [ - 1) and x in the order of L. Notice that if x < y are 
relevant, then either ,J  Ju =  or & C ,lu (in which case we also bave Il  Ixl, 
:F(lyl- 1) = yF(M- 1) and .r(ly]- 1)  y(lyl- 1)). 
Define Z = Z() to be the collection of all finite unions of sets of the fornl 
• LforaGA0, 
• ci(&) for relevant x, 
• cl({x}) for x Ç LH«h, and 
• Lœ  Lad for x Ç LHch. 
So L() is a subtree of (* U) < (i.e., it is closed under taking iifitial segments). 
The nodes belonging to Lmad are exactly the terlnilml (= maximal) nodes of this 
tree. The point of the Jx is that we need "copies" of the large supports given by 
the L for isomorphismf-names arguInents. The S, then, are used to code the 
places where we put the .J so that we basically get well-foundedness for free. 

Lemma 2.1. (L,Z) is a ternplate. 



2644 j01ï{G BIRENDLE 

Proof. Clauses (1) and (6) in the definition of template are immediate, as is closure 
undcr finite rotions. To see closure under finite iutersections, it suffices to argue 
that the intersectiou of any two sets of the above form (i.e., Le, ci(Je), d({x}), 
and L  Ld) is again of this form. This, however, is straightforward so that (2) 
holds as well. 
To prove (3'), lct x G L«h and g G L,. In case g G LH«h, we have g G c({g}) 
For (4), it suffices again to consider sets A from Z of the above form. Let 
x G L«h 4. Without loss ofgencrality. AL« ¢ 0. If 4 is ofthe form L 
d(Ju), cl({g}) or L u  Ld, then we must have V > x. A = L u is impossible and 
if A = cl(Ju), then x < g[(]g]- 1) = min(Ju). So, in each of the possible cases, the 
intersection with L, is L,  Lmd- 
 are left with showing well-foundedness (5). Assume A,,., n G , is a de- 
crcasing chain ri'oto Z[LH«h. Let a be such that Le,  LH«h occurs in A, as a 
component. Choose a, o milfimal among thc a. Without loss of generality, n0 = 0. 
Then all Le,  Lch are thc saine and it suffices to consider the J«-components. 
Thus we may assume, without loss of generality, that A0 =   LH«h, and there 
is a finitely i)ran«hing tree T ç w < such that A,, = («¢Tm- "]x U Fn)  LHe«h 
where the #Ç ç LHch are finite, and such that a Ç , la I =  < I1 = m. implies 
.1 Ç ,1, and such that the .1«, a G T  , are pairwise disjoint. Now note 
tlmt if f G [T] is a bra.nch, t/mn the sequence {.rI"; ,, G w} must eventually sta- 
n} wouhI constitute a decreasing sequence of ordinals. Then notice that if Ix{ 
is eventually constant, so is the decreasing sequence x{I'(]x I] - 1).) Since T is a 
finitely branching tree this means that the total number of the x is finite which 
in turn entails that the sequence of the .4, eventuallv stabilizes. 
Corollary 2.2 (Bounds for a). FIL forces b = 0 = Ao and adjoins a mad familv 
of size  (so that o  a   ). 
Proof. This is immediate bv construction of FIL and Propositions 1.6 and 1.7. 

3. KILLING MAD FAMILLES USING TEMPLATES 
We are left with showing there is no mad familv of size legs than A in the 
generic extension. As explained in the Introduction. this is an (albeit sophisticated) 
isomorphism of-names argument. Isomorphisms of names canonicallv boil down to 
certain brands of partial isomorphisms between subsets of L. and we begin with 
their investigation. 
Definition (Isomorphism). Let A.B C_ L be countable trees, s Call A and B 
isomorphic (.4  B) iff there is a bijection ¢ = CA,U" A --, B such that 
(a) I(«)1 = I1, 
(b) 4(«)In = 
(c) « < y iff re(x) < re(g), 
(d) x(n) is positive iff ¢(x)(n) is positive, 
(e) x G Ld iff ¢(x) G Ld 
for all x, g G A and all  G w, and such that 

8F{ecall A is a tree if it is closed under taking initial segments, i.e., given x Ç A. we have 
x[nÇ,4 for ail n Çw. 



THE ALMOST-DISJOINTNESS NUMBER 2645 

(f) Z[A is nml)pcd to TIB via 
Since the trace of Z on each comltable set is comltable, there are at most 2 ° = 
isoinorphism types. 
This, the strongest notion of "isomorphism" we shall consider, vill be used in 
several pruning arguments below. However. tbr most purposes the following is 
sutiicient. 

Definition (Veak isomorphism). Let A./3 C_ L be a.r|fitrary. \Ve sav that .4 and 
/3 are weakly isomorphic (.4 wk /3) if (e) is satisfied and instead of clauses (c), 
(f) we have 
(c') x < y iff ff(x) < ff(y) fol" all .r.y sueh that there is z G LHhçA with 
x<_z<_y, 
alld 
(ff) ff lllaps a cofinal SIlbSCt of Z[A to a cofilla.1 subsct of 
respectively. 
Lemma 3.1. Let 4 md t3 b« countable trees such that LHch ç A (L«ch  t3, 
respectiv«l.q) i,s cofinal i, A (in t3, resp.). If A - 13. as witnessed by ff), then there 
is l, ectending ff and wit.nessing that ci(A) wk ci(B). 
Pro@ Call a nonempty X C ci(A) V1 Lmad counected if given x < y ffoto X, the 
interval Ix, y] is disjoint ffoto A ç LHch. A maximal coimected set is called a 
connected comportent. Note every commcted COlnponent has size A (because LHech 
is cofilml in 4) and ci(A) ç? L,nad is a disjoint union of at lnost comtablv man.v 
connected components. 
Given x  LHechç?A with .r([xl-1) being positive, put Comp. r 
z for every z  A ç? LHech with z D x and y > z for every z  A ç? LHech such that 
z[[x I < x}. Clearly Comp« is a commcted colnponent. Duall.v, define Comp.,. for 
x  LHech ç? A with negative x([z I - 1) by interchanging < and >. 
For each y  ci(A) ç Lmad. there is x  LHech f-I A with y  Comp. To see this, 
let n < [y[ be lnaxilnal such that y[n  A. Assume. without loss of generality, 
is positive. Let k <_ '« be minimal such that all y(i) for k _< i _< n are positive. 
If possible choose m, k _< m < ,, and x  A çl LHech , Ix[ ; .l ÷ 1, such that 
xlrn = ylm, x(m) > y(m) is lninimal, and such that m is the maxinml value for 
which such an x can be round. Then y  Comp«. If m and x camlot be round, we 
let x = ylk and check y  Comp.Æ (note that (Ixl- 1) ; .v(- 1) is negative in this 
case so that the second alternative of the definition of Comp applies). Therefore 
ci(A) Ç1 L,nad = UzLHechA Cmp.. 
Also notice that for x.x   LHech çl A, if Comp« = Comp,, then Comp¢(x ) = 
Comp¢(x,), and if Comp« ç Comp, = 0, then Comp¢(x 
So we can simply extend ff to g, by mapping Comp. to Comp¢(x ) for all x 
LHech  A. Then (c ) and (e) are ilnlnediate. To see (f), note that bv defiifition 
of the template, sers in Z[cI(A) that are Ulfions of sers ff'oto Z[A and of sers of the 
form L V1Lmad are cofinal in :/TIcl(A ). However, since ff identifies sers of ZIA and 
sets of Z[B. b identifies sers of the latter kind. [] 
Note that we did hot use the full st.rength of out notion of isolnorphism in the 
above proof. Clauses (c) and (f) could be replaced by (d) and (f) respectively. 
Furtherlnore, instead of dealing with tlees A and/3 (and having (a), (b), and (d)), 



2646 .IFIG BIRENDLE 

it suffices that ci(A) çl Lmad is the ullion of the COlni)onents Comp., x E LHech A A. 
and similarlv for/3, and that extending ¢ by lnapping Cmp. to Comp¢(z) preserves 
Lemma 3.2. If A w«k B. the,, PI,4  
Pro@ Notice that clauses (c), (e) a.nd (f) are enough to guarantee that 
D[B. Bv Lelmna 1.5, this is still true if (f) is replaced bv (g). Finally, bv the 
cas" PI,4 is defined recursively, interchalging elelnents of Lmad that belong to the 
saine COlmected COlnponent of 4  Lmad does hot affect the p.o. 9 (because the 
interchalging mai) sends a cofilm.l subset of Z[A to a cofinal subset of Z[A. see 1.5). 
Completion of ttte proof qf the kIaiz Tteorem. Now assume À is a naine for an 
ahnost-disjoint familv of size < A, say À is listed a.s {,4; a < h'}. Also assulne 
is forced to have size at least A0. Let k <  be maximal such that ,  Ai.. Without 
loss of generality, h  A-. 2. 2, shall perform several standard pruning argulnents, 
reordel'ing the family of the À' so that the first A many look very "silnilar", that 
is, those  that do llt fit the t)att«ll get relnoved to higher indices. This is whv 
we stilmla.te h  A- 2. Eventually. the first  lllallV Àa will suffice, and it is those 
that we use to create a lleW llallle ,h «itnessing non-lnaximality. 
For fixed a, fiud c«mntablc maximal anti«hains {P,i; i G } ç 
tlid {,i e 9. i, , e w'} Sllch that  4" 
Pmi lb   . iff .i = 1 and 
iffk.i = 0. Let B  = 0{dom(p.i): i.n  } ç L. B  is at most countable. 
Without loss of generality, it is a tïe. Let C  = cl(B). Put B :=  B . So 
 < + < A. Bv CH and the  systeln lemlna we inay SUlne, without loss of 
generality, that tho {; o < N} forln a -systm. ald that th bijction  = ' 
(see above) sending B fo B is an isolnorphisln fixing the roof R of the svstem. 
Beca.use t here are only Aï i =  - i mmv COllt able subset s of L  h  (* U A) . we 
may also assume that if' Lech and   '. then z  . Also stipulate that 
thre is SOlne 00 < 'i such that whenevr o < Nh, •  , j odd md (j) 
then (j)  N for SOlne 0 < 00. As xplaind above,  cmonically inducs a 
isolnorphisln  = @' betwe«n C  and C  (Lemlna g.l), which in turn yiIds an 
isolnorphism ç = i a, bet.ween PC  md PIC  (Lelmna g.2) both of which elnbed 
iuto FL (Main Lelmna i.I), as well as between FC-imlnS ld [C-naluS. 
Furthernore, since connected components are holnogeneous froln the forcing point 
of view, since C   L«h = B  L«h is countable, and since C   Lmad h only 
countably inany COlmeCt.d components (see th proofs of Lemlnas 3.1 and g.2), it 
bas, up fo isomorphisln, only 2 ° = N inanv isolnorphisln types of nalneS. (Of 
course, there are a total of A ° nalneS.) Therefore, we lllV g]so suppose that 
identifies  with .. which in,ans, more xplietly. tlmt ,i := '.i = ' md 
p 
XçS'ite B  = {z: s  T} where T ç ( Ul) < is the canonical tree isolnorphic 
to anv B . This means in particular that '(z) = «, that I1 = I1, and 
that s(z) is positive iff z(n) is positive. Let THech = {$  T:   LHeeh} and 
Tmd = {s  T; z  Lm }. Tn«h is a subtree of T, while Tmd is a set of terlninal 
nodes of T. Furtherlnore, let S ç T be the subtree of T corresponding to the root, 

9That is, the order structure on connected components of .4f3 Lma d is irrelevant, and connected 
components are homogeneous from the forcing point of view. 



THE ALMOST-DISJOINTNESS NUIVIBER 2647 

a a xs  iff s E .9. Furthermore, 
that is, sE Siffx» G R for allo. So. foro ¢/3, m s = 
if s G THech  S. then [s I OE k + 1. List {t G T S; t[([t[- 1) G S} = {t.; n k 1}. 
For a < fl define 
{ mill{n; ei{her t.([t,,[- 1) G w and x a X  
F({,}) or t(l/l- 1)  w alld a'a - - 
= t.(If. I 1) <a-.(It I 1)} 
if such ail it exists, 
0 otlwrwise. 
Note tha.t for each n k 1, eVely subset of Ai. honlogeneOllS in color , lll/ISt be finite. 
Using partition calculus as well  standard pruning argtllnelltS, we nlay therefore 
assmne that for all a < Wl, if s  S and s{Ç)  S. then 
• if ç is positive, then 
-(ç)(s]) for all  < , and all x  
are larger than Wl, and if Ç < , s{Ç),s{)  S, then 
- either for ail a, fi, we have 
(this is t.he case when sup z(ç)(s) < sup ()(lsl)), 
-- or for ail ( < 
x.<>(Isl) < • <ç>(Isl) 
(this is tllc cse whPn sllpa 
• if Ç is llegative, then x-(ç)(sl) > x  
S'(Ç),S'{}  ,ç, t]len 
- either for ail a, 
- or for ail a < , we have x a 
Define x s  L by recursion on the length of s  T  follows. If s  S, then x s = x s 
(a fortiori Ix;] = Ixl = Isl). If s  S lld s'<0  Ha  S, 1Pt x-«>(l*l) be th 
linfit of the x-<ç>(ll) (so it is either the sup or the inf, depending on whether Ç is 
. 0o 
positive or negative). Next find % < ll+l,  > Wl and s  SII+I, such that for 
all such s and Ç, 
• if xç.<o(Isl) = sup.x.<o(Isl), thon for ail 
• if x2.<o(Isl) = iIlf. Xs(ç)(Isl), then for 
-«> r(Isl + 1), we hv y(Isl + 1) < %. 
It is clear that we Call find such s's because ls[+l > [BI is regular (since [si k k). 
To complete the definition of xs-(ç ) stipulate I s-/01 = Ix-<ç>l + o I*l + 3, and 
define 
{x 
,-(ç)([s[) if [si > 0, 
a--{ç)(Is I + 2) = ç if Isl = 0. 
If s  S and s'(ç)  Tmd S, find 7  A* OA such tiret ()  Lmd and 
for ail   B with [IsI = ., we have (Isl) # . Such  clearly exists because 
A > IBI . Stipulate Ia.-{ç)l = I .-{ç)l = Isl + 1 and let a.-{ç)(Isl) = 7- Finallv. for 
the l'emaining t  T, stipulate again laîl = laPl + 2 = Itl + 2. find s c t with s  S 
maximal, and ptt aî[(Is ] + a) = a and a'î( + 2) = m() for  > Is I. 
Let B  ={.'., s T}. Notice t, hat B , although verv trelike, is not a tree 
like the B's. We proceed to verify that (B.gB ) and (B.g[B), a < w. are 
weakly isomorphic (clearly (a), (b) and (d) will rail but this is hot relevant for us). 



2648 .IIRG B1RENDLE 

• a ' \Vithout loss of generalitv, o = O. (c') 
Fix o and define ¢ = ¢" by ¢(a ) = x. . 
and (e) are ilnmediate by construction. So let us check that the trace of  on B ° is 
lnapped to a cofilml subset of [B . First fix  and consider L. Notice that there 
.o  L 
is ff0 5  such that ç(L&B °) = L&B  = LB . For any s  T with 
 . (0) > 
vet x s ¢ LB one lnust have a (0) > /5  0 and x(0) is an oe-lilnit of 
the x(0)'s. In particular, for ail such s. x(0) nmst have the saine value, say 70- 
Also .r(1) = 70 and «(2) = s(0) < w. This lneans, however, that L3 B ° is 
lnapped to (LBO cl(a«)) B via ç whore ].ri : 3, x(0) : 70, x(1) : 7) and 
x(2) : sllp{.s(0) + 1; x(0) < Ç}. Next assume a" is relevant and considor 
0 «r(ll-). 
Assulne &B ° ¢ . Then there is s  T such that [s = Ix I- 1 and x s = 
" r ° and ,le  B ° is mapped to ,& B  via ç because 
In case s  S, we have ms = • s • 
0 l,a v(ll ) < (11- ). 
z e n for l,y  e g0 with I1 = *, r(ll- ) = « - - 
[11 Ce 8  T  S, let j0 ( 18] lle lnaxi,nal with sU0 e S. Defilm g by 
g[([ - 1) : d s "n and g(]l - 1) : x(Ix ] - 1) and note that .1«.  B ° gets mapped to 
u  B  provided wc can show  is relewmt. In case jo > 0 there is nothing to show 
lmcause whelmver 0 •  =*(3) > w. and. 
a'(3) > wt. J  j0 even, tht'n also .r s(j + 2) 0 - 
. " " (j0) > w while, if J0 is odd. 
if j0 is even. we additiolmllv have a s (30) = SUl)a xs 
we additionally lmve s(3o + 1) = 7sjo > w. In case j0 = 0 this is true because 
.,'(1)  Sï  and 00 is larger than ail the 0 for which a'g(j)  S where j > 1 is oaa. 
Even though B  is not a ri'ce, xve can verify, as in the proof of Lelnlna 3.1. that. 
letting Ç := cl(B), Ç  Lmad = ,eT Comp, and that ç = ' : Ça + C, 
et < w, which extends ç and lnaps Comp« to Comp, is a weak isonorphisln. 
B)- Lelnlna 3.2, ?[(' and F[C are isomorphic bv a map  = '. X sends 
[Ca-lmlnes to [C-nalneS, and we defilm .n to be the image of .a under 
Bx- constructioll, it is then also ilnllmdiate that whenever  < , we can find 
o < Wl such that B  U B  and B a U 3 are weakly isolnorphic via the lnapping 
fixing nodes of B  and sending the x to the corlespondilg x, and such that 
this lnapping identifies cofinal subsets of the traces of Z on the two sets. m Again. 
this weak isomorphism canolficallv extends to a weak isomorphisln of C U Ç and 
C a U C, which in turn means that F[C U C  and F[Ca U C  are isolnorphic 
(Lemlna 3.2) bv a mapping sending the llalne .4 to a. Since À  and . are 
forced to be ahnost disjoint (by F[Ca U C), so are  and . ' (by the iSOlnorphic 
[C  U Cç). Since [C  U C embeds into [L (Lelnlna 1.1), this completes the 
proof of the non nmxinmlity of  and, by Çorollary 2.2. of the Main Theoreln. 

[:{ EFEIENCES 

[BJ] T. Bartoszyfiski and H. Judah, Set Theory. On the structure of the real line, A Ix Peters, 
\Vellesley, MA. 1995. MR 96k:03002 
[B1] A. Blass, Combinatorial cardinal characteristics of the continuum, in: Handbook of Set 
Theory (A. Kanamori et al., eds.), to appear. 
[Br] J. Brendle, Mad families and iteration theory, in: Logic and Algebra (. Zhang, ed.), 
Contemp. Math. 302 (2002), Amer. Math. Soc., Providence, RI. 1-31. 
[H1] S. Hechler, Short complete nested sequences in N \ N and small maximal almost-disjoint 
familles, General Topology and Appl. 2 (1972), 139-149. MR 46:7028 
[H2] S. Hechler, On the existence of certain cofinal subsets of c0 °. in: Axiomatic Set Theory (T. 
Jech, ed.), Proc. Sympos. Pure Math. 13 (1974), 155-173. MR 50:12716 
[MI A. Millet, Amie Miller's problem list, in: Set Theory of the Reals (H. Judah. ed.), Israel 
Math. Conf. Proc. 6 (1993). 645-654. MR 94m:03073 

10In fact. this is true for ail but countably nlany 0 < Wl. 



THE ALMOST-DIS.IOINTNESS NUMBEIR 2649 

[SI] S. Shelah, Covemng of the null ideal may have countable cojïnality, Fund. iklath. 166 
109-136. (publication nmnber 592) iklR 2001m:03101 
[$2] S. Shelah, Are  and ] your cup of tea? Acta Math., to appear. (publication number 700) 
[vD] E. van Douwen, The itegers and topology, in: Handbook of Set-theoretic Topology (K. 
Kunen and J. Vaughan, eds.), North Holland, Amsterdam (1984), 111-167. iklR 87f:54008 
IV] 3. E. Vaughan, Small uncountable cardinals and topology, in: Opea Problems in Topology 
(.J. van Mill and G. ikl. Rced, eds.), North-Holland (199(}), 195-218. 

THE (]RADUATE SCHOOL OF SCIENCE AND TECHNOLO(;Y. KOBE [INIVFISIT'. I{)KKO D3,1 1 1. 
NADA-KU. KOBE 657-8501, .I,PAN 
E-mail address: brendle@kurt, scitec, kobe-u, ac. jp 



TRANSACTIONS OF THE 
AMERICAN MATttEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2651 2662 
S 0002-9947103)03283-5 
Article electronically published on March 14, 2003 

CYCLICITY OF CM ELLIPTIC CURVES MODULO p 

ALINA CARMEN CO.IOCAIU 

ABSTRACT. Let E be an elliptic curve defined over Q and with complex multi- 
plication. For a prime p of good reduction, let. Ê be the reduction of E modulo 
p. We find the density of the primes p <_ oe for which E(Fp) is a cyclic group. 
An asymptotic formula for these primes had been obtained conditionally by 
J.-P. Serre in 1976, and unconditionally by Rare Murty in 1979. The aim 
of this paper is to give a new simpler unconditional proof of this asymptotic 
formula and also to provide explicit error terres in the formula. 

1. INTRODUCTION 
Let E be an elliptic curve defined over Q and of conductor N. By a famous result 
of Mordell, the set E(Q) of Q-rational points of E is a finitely generated abelian 
group. The study of the free part of E(Q) is still one of the major problems in 
arithmetic geometry. 
Now, for a prime p of good reduction for E (that is, p  N), we denote bv E 
the reduction of E modulo p. This is an clliptic curve defined over Fp, the finite 
field with p elements. Naturally, as in the rational case, one is interested in the 
study of the structure of the group E(p) of Fp-rational points of E. From classical 
theory, E(Fp) can be written as the product of two cyclic finite groups. Indeed, 
E(Fp) C_ E(Fp)[k] C_ Z/kZ Z/kZ, where Fp denotes the algebraic closure of Fp, k 
is a positive integer such that the order #E(Fp) of E(Fp) divides k, and E(Fp)[k] 
denotes the group of Fp-rational points of E annihilated by k. Early computations 
of Borosh, Moreno and Porta ([BMP]) showed that, in fact. for "many" primes p. 
the group E(]p) is cyclic. One expects this to be true for infinitely many primes 
p, as suggested by the elliptic curve analogue of Artin's primitive foot conjecture 
formulated by Lang and Trotter in 1977 (see [LT2]). 
Out goal in this paper is to provide an asymptotic formula, with explicit error 
terres, for the function 
f(x,Q) := #{p <_ x " p ,,E(Fp) cyclic}. 
in the case of an elliptic curve E defined over ( and with complex multiplication. 
In 1976 (see [Sel]), J. -P. Serre showed that C. Hooley's conditional method 
of proving Artin's conjecture on primitive roots (see [Ho, ch. 3]) can be adapted 
to estimate f(x, (). More precisely, let ( denote the algebraic closure of ( and 

Received by the editors July 24, 2002 and, in revised form, December 4, 2002. 
2000 Mathematics Subject Classification. Primary 11G05; Secondary 11N36, 11G15, 11R45. 
Key words and phrases. Cyclicity of elliptic curves modulo p, complex multiplication, appli- 
cations of sieve methods. 
Research partially supported by an Ontario Graduate Scholarship. 

9K1. 

(2003 American Mathematical Societv 



2652 ALINA CARMEN CO IOCARU 

let Q(E[k]) denote the fiel(l obtain(,d by adjoining to Q the coordinates of the 
Q--rational points of E7 annihilated by k. T]lel, under the Generalized ienann 
Hypothesis (denotcd GRH) tbr the Dedekind zeta ftmctions of the division fields 
Q(E[k]) of E, Serre proves that, as ,r + OE, 
(1) f(x,Q) = fli,c + o . 
where li,r := f" 1 dt is the logarithmi« integral and 
k--I 
with p(-) denoting the M6bius fmlclion. 
X> recall hal for real-valued fimctions f and 9 # 0 we write f(z) = o(9(«)) to 
lllean that lim. I(*} _ O. Also. if 9 has p,,sitive values, we write 
g(x) -- 
()r f « g fo mean that there exists a 1)ositive constant 
Ag(x) Vx. If lhe COllSlallt ,4 d('i)ends on S()llle qnantity B. then we may write 
/(x) = ()(9(x)) or f « g. In this paper, wllenever we write f(x) = O(g(x)) or 
.f « 9, we nlean thal thc imi)lied O-constants are absolute. If f « g « f, then we 
write f  9. 
In 1979  (sec [Mul. pp. 161-167]), Rare Murtv removed GRH in formula (1) 
tbr elliptic curves with complex multil)lication (deuoted CM). His proof uses clss 
fiekl theoretical properties of the division fields of CM elliptic curves, as well as a 
number fiekl version of the Bombieri-Vinogradov theorem (whose proof is based on 
the large sieve tbr number fields), lu 20(10 (sec [acC1]), the author proved formula 
(1) for elliptic curves without complex multiplication (denoted non-CM) under the 
assumption of a quasi-GRH (more precisely, a zero-free region of real part > 3/4 
for the Dedekind zeta functions of Q(E[:])). For more history about /(x,Q) in 
i)oth the CM and non-CM cases we refer the reader to [acC1], [aeC2] and [Mu3]. 
In this paper we give a llPW simpler unconditional proof for the asymptotic 
formula for f(x, Q) in the complex multiplication case, and provide explicit error 
terlns in this fornmla. ç are proving the following: 
Theorem 1.1. Let E be a CM elliplic curve d«fin«d over Q, of condctor N ad 
with complex multiplicatio by the full tin9 of integers 0- of an imaginaw quadratic 
fi'eld Iç = Q ( ) . wh«re D is a positive square-free znt«ger. Then, as x 
(2) f(x,Q)=IElix+O.v (logx)(logloglogx) " 
or, more precisely, 
(3) f(x,Q)=IElix+O (log x)(log log og,  1Og log x . 
where the O-constant in (2) depends on N and the one in (3) is absolute. 
Corollary 1.2. Let E be a CM elliptic curve defined over Q, of conductor N and 
sch that Q(E[2]) ¢ Q. Then the smallest pme p { N for which Ê(Fp) is cyclic 
. ,« o (xp () ) . r v«e O-co.  

lit, was communicated to the author by Rare Murty that this result was obtained in 1979; 
however, it appeared in print onl3r in 1983. 



CYCLICITY OF CM ELLIPTIC CURVES MODULO p 2653 

It is possible that the error t.erlllS iii Theorenl 1.1 Call be ilnproved, but this 
involves lnore sophisticated lnethods than the ones used iii our pal)er. \Ve relegate 
this te ftltllle research. 

2. PRELIMINARIES 
2.1. Notation. Given ai1 elliptic curve Æ defined over Q, p will always denote a 
prime of good reduction for ?. \Ve set ap := p + 1 - E(IFp) and say that p is of 
ordilmrv reduction if ap  (, ami of supersilgular reduction if ap = 0. We denote 
bv zcp and  the roots of t|le po|ynomial ,\' - aw\" + p  Z[X]. 
If hot otherwise stated, q will denote a rational prime and k a positive integer: 
n(x) wi|l denote t|le nmnber of ratiollal prilnes G ,r; S will denote the cardinality 
of a set S: Ker ç wil| denote the kernel of a morphism ç. 
2.2. Algebraie prelinfinaries, The followmg prelilninary |enmlas are well kmwn. 
but, for the sake of completeness, we inçlude thenl hel'e. 
Lemma 2.1. Let E be an elliptic curve defined over Q and of condu«'tor N. Let 
E[k] be the gro'u 1, of k-divi.sion points of E. Then 
(1) the ramified p,'in, es oJ'Q(E[k])/Q are divisors of hN; 
(2) assuming that E bas co'mplex multiplication and !« > "2. we hat,e 
O(k) " << [Q(E[k])-QI << k 2, 
where çb(k) dezotes the Euler flm«tiou. 
For proofs of this lelnlna the reader is referred te [Silvl p. 17"9] and [Sllv_. 
p. 1351. 
Lemma 2.2. Let E be an elliptic curve defined over Q and of conductor N. Usin9 
the notation introduced in Section 2.1 we bave that, for a positive izde.qer !; and a 
prime p !; of 9ood redu«tion for E, 1» splits completely in Q(E[k])/Q if and only if 
zrp-- 1 
 is an algebraic integer. 
Pro@ \Ve recall that 7rp is the algebraic quadratic integer corresponding te the 
Frobenills endonlorphislll 
which we also denote by 
Since (p, kN) = 1, we have that 1' is ulralnified in Q(E[k])/Q (see part 1 of 
Lemna 2.1). By classical results in algebraic nulnber theory, p splits completely in 
Q(E[k])/Q if and only if 7rp]oe[] = 1, where 1 denotes t.he identity map. This last 
condition is equivalent te saying that Ker([h]) C Içer(wp- 1) as nmps Ê (-7)  
Ê (-7), where [k] is the multiplicatiol, by k map. Hence there exists an elliptic 
clIrve endomorphism b" E (IFp)  Ê (-7) such that b o [k] = rrp - 1 (see [Silvl. 
Crolla.ry 4.11. p. 77]). This is equivalelt to saying that '- is ail a.lgebraic 
k 
integer. [] 
Lemma 2.3. Let E be a CM elliptic curve defined over  an.d with complex mul- 
tiplication by an imagincwy quadratic field Iç. Then. for every prince p of ordinary 
good reduction for E, we have Q(Trp) = K. 



2654 ALINA CARMEN COJOCARU 

Pro@ First we observe that 
Q(rp) c_ Endp(E) ®z Q c_ Endw(E ) ®z Q. 
Then we note that, since E has complex multiplication by/x, we have an embedding 
K C Endw(Ê ) ®z Q, and, moreover, since p is a prime of ordinary reduction, we 
actuallv have K = Endw(E ) ®z Q. Thus Q(rrp) c K for any prime p of ordinary 
reduction for E. But K is a degree 2 extension of Q, and so is Q(rp). This gives us 
the desired equality. [] 

Lenlma 2.3 describes a feature of Chi elliptic curves that will play a very impor- 
tant role in our unconditional estimates of f(x, Q). It actually describes one of the 
main differences between Chi and non-CM elliptic curves (see [LT1]). 

2.3. Analytic prelilninaries. The next prelininary lemma is an application of 
the sieve of Eratosthenes, which we recall below. 

Theorem 2.4 (The sieve of Eratosthenes). Let .A be a set of natural nrnbers <_ x. 
and let 7 ) be a set of rational primes. To each pmme p 
distinguished residue classes modulo p. For any square-free integer d composed of 
primes of 7) we set 

.A(d) := {a  .A • a belogs to at least one of the w(p) residue classes 
modulo p for all 

and 

() := 1-[ (P)" 
p[d 
For a fixed real number z, we let S(.A. 7), z) be the number of elements a  .A that do 
not belong to any of the distinguished residue classes modulo p for all p  7), p <_ z. 
and we set 

We assume that 

(1) there exists a real number X such that, for all square-free integers d com- 
posed of primes of 

(2) 

#A(d) = X- + Rd 

for some Rd = O(w(d)); 
 w(p)logp _< clog z + O(1) for some positive constant c 
P 
pP 
p<_z 

Then 
S(.A,7),z) = XI"(z) + O (x(log z)C+l exp (--- 
where the implied O-constant is absolute. 

log 

For a proof of this result, see [Mu4, p. 141]. 



C CLICITY OF Chi ELLIPTIC CURVES MODULO p 2655 

Lemma 2.5. Let x • ]R and let D. k be fixed positive integers with k < v @ - 1. 
Then 

Sî := # p<_:r'p= k+l +D l'2forsomect,/3•Z 
: O((-.X+l) v/71°gl°gx ) 
kv log --: --A " 
The implied O-constants are absolute. 
Proof. 1. Let us observe that the conditions p  x and p = (ok + 1) 2 + D/32k 2 for 
some a, fl G Z imply 

Thus 
() s _< }2'# { 
where the sure '¢ 

-1 - x/Tl`. , -1 + v/7]l`: 
V V 
»v' 'v 

çZ, 

nz, #0. 

• [ -1-v'@k '--l+v@]k rZ'(ah+l)2+D/3212apri'ne}" 

a E k ' k flZ , 

7) := {pa rational prime" (p,k) = l, (pD) = l}, 
with (}) denoting the Legendre sylnbol nlodulo p. To each prinle p • 7) 
we 
asso- 
ciate the residue classes 
(-1 + [3kD)k -1 (modp), 
where D is an integer such that D 2 = -D(modp) (let us observe that (ck + 1) 2 + 
D132q 2 = p imposes the conditions (=) = 1 and (p,k) = 1. and hence D and 
k-(modp) are well defined). 
For a fixed real number z we have 
 oe• . ,  V1Z'(at,'+l +D/32k 2aprime 
(5) < s(A,V,z) + ,(z) 
< S(A, 7), z) + z, 
with S(A, 7 ), z) defined as in the sieve of Eratosthenes. 



2656 ALINA CARMEN COJOCARU 

Now we want to verifv that the hyl)otheses of Theorem 2.4 are sa.tisfied. Ele- 
mentarv estimates givc us 
#.A(d):=#{ce.A'(ok+l)2+D1321«2=O(nmde)}=2 +1 
for ail square-free integers d composed of primes of P. Thus the first hypothesis 
of the sieve of Era.tosthenes is satisfied with w(d) = 9 and X =  
-  + 1. Using 
glertens" theorem and recalling tha.t () = 1. hence that p splits completely in 
 ,(v) ov _   og, _g: + o(1). 
vz pz 
Tlms hc sccoud hyl»ohcsis of lhe sicvc of Eroshcnes is stisfied wih c = 1. 
Thercfore, 
+l ll'(z)+O  +l(logz) 2exp 
k log z " 

Il {z) = 1 - _< exp -2 « exp(-loglogz)- loz' 

by using the elemeutarv inequality 1 + t <_ exp(t) and. a.gain, Mertens theorem. 
Let us choose z such that 

Then 
O(X/-t- 1 
k 
and so 

s(A. p. z) 

log z -- 

3 log log .r" 

(log log x  
: (--. -t- 1) O \,og  j 
-- (+,)o. 

V+I 1 ) 
k log.r(lo log x) 2 " 

(  ) 
+ 0 k log x(log log x) 2 

From (5) we obtain 

= (. + 1) 0 ( l°-g l°g'r log __2!_._ ) , 

which, used in (4), completes the proof of the first part of the lemma. 
2. Sinfilar to the proof above 



CYCLICITY OF CM ELLIPTIC CURVES MODULO p 2657 

We relnm'k t.hat for SI. and Sî. of the al)ove lelnlna we aetuallv have elelnentary 
log log  
estimates that are weaker than t.he ones given bv Lelnlna 2.5 onlv by a 
log x 
factor. The sieve has been invoked pre«isely fol" obtailfing this saving. 

Lemma 2.6. Keeping the notation of Lemma 2.5, we bave that, for any k and x, 

, () 
t. «< /,.-- +1 , 

where 1 <i<2. 

Pro@ We justify this estilnate for i = 1. The case i = 2 is l'esolved similarly. We 
observe hat the Colditions p <_ .r and p = (ak + 1) 2 + D/32k 2 for some a,/3 G Z 
give us - + 1 choices for o raid  choices for/3. The lemlna follows. [] 

3. THE PROOF OF THE THEOREM AND COROI,LARY 

As explained in [Mul. I,I»- 153-154], we have that E(IF») is cyclic if and only if 
p does not split COlnpletely in Q(E[q]) for any prime q ¢ p. Also, we have that if 
p _< .r and p splits eompletely in Q(E[k]) for SOlne /,', t|len k21(p + 1 - ap), and so, 
using Hasse's bolnld ap <_ 2V@, we obtain k _< 2v. Therefore, using the simple 
asylnptotic sieve, we tan write 

fez, Q) = N(x,g) + 0 (lll (x,y, 2v#-) ) , 

where 

]V(z,y) := 4¢ {P _< x" pdoes llOt split COlnpletely iii ally Q(E[q])/Q,q <_ y}, 

11I (x,v. 2v') := # {p < x" psplits conlpletely in someQ(E[q])/Q 
with .q < q < 2V}. 
and where y is a real nulnber to be chosen later, hl order to estinlate f(x, Q) we 
need to estimate ea«h of N(x, y) and ]Il(x, y. 2v ) and to «hoose the paralneter y 
appropriately. 

3.1. Estimate for N(x, y). By the inchlsion-exchlsion principle we have 
]V('F,V) = Et/-/(/')71-1( x" Q([/çl)/Q), 
where the sure is over ail square-free positive integers k < 2v whose prime divisors 
are < y, and where 
rl(z, Q( E[k]) /Q) := #{p < x "p splits COlllpletely in Q(E[k])/Q}. 
We estimate this sum by using the unconditional effective version of the Chebotarev 
density theorem as stated in [Mu2, p. 243] or [acC1. p. 337]. To do so, let us recall 
froln [Se2, p. 130] that if L/Q is a finite normal field extension that is ramified only 
at the primes p, p2,..., Pr.. then 
IL" Q-- l°g I disc(L/Q)l -< log[L" Q] + E logpj, 



2658 ALINA CARMEN CO.IOCARU 

where [L'] and disc(L/) denote the degree and the discrilninal,t, respectively, 
of L/. V'e apply this result, together with Lemma 2.1, to the fields (E?[k]), whose 
degree and discrinfinant over  we denote by ri(k) and d, respectively. D,'e get 
n(k)ldkl ' ŒEE k8N 2 

and 

,,(k) 00g 141) 2 « k6 00g (»2v))2, 
and so the maximum of the two quantities above is « k8N 2. In order to apply the 
mmonditional effective Chebotarev density theoreln mentioned before we need to 
have kN  « log x. Since k  exp(2y), it is enough to choose 
(6) y = (log log x - 2 log N). 
Then, by the unconditional effective Chcbotarev density theorem, we obtain 
N(x, y) = n(k)] lix+ O a'exp -A 
tbr some effective positive constant A. To handle the error terre we use that n(k) « 
k 2 and that there m'e at lnost 2 .v square-free numbers composed of primes  y. Then 
(r) N(x,y)= ç n(k)]lix+O N/4(logx) B 
for any positive constant B. 
3.2. Estimate for M (x, y. 2). For rem numbers {, {2, we denote by 
ao («.{.{) 
the number of primes p  x such that p bas ordilmry reduction and splits completelv 
in some Q(E[q]) with {  q  {2, and by 
M(x.{.{) 
the lmmber of primes p  x such that p has supersingular reduction and splits 
completelv in some Q(E[q]) with {  q  {2- X write 
(s) M (, , ) = a o (x.v. ) + a * (,. v. e) 
and estimate each of the two terres. For the first one we observe that 

(9) 

where 

y<q_<2q 

rcï(x,Q(E[q])/Q) := #{p <_ x" ap J: 0 and p splits completely in Q(E[q])/Q}. 

By Lenmms 2.2 and 2.3 we obtain 
,ï(x,Q(F[q])/Q) < # {p < 
Since the norm of rp in K/Q is p, we get 
{ p -- 1 
# px. 
q 

X ° -- 

rcp - 1 " 
 (DI,- . 
q 



CYCLICITY OF CM ELLIPTIC CURVES MODULO p 2659 

where Sq is Sq  if-D = 2,3(mod4), and ,çq if-D = l(mod4), with Sq, Sq 2 as in 
Lelnma 2.5. 
Let us fix a real nulnber "u < vif- 1. Using the elementary estinmte for Sq given 
in Lelnnm 2.6, we obtain 

Z rï(.r, Q(E[q])/Q) <_ 

(10) << 

 Sq 
  (7 1) 
<q<2/ qv + 

2x 1 v 
u<q<_2x/' 
x  log log a" 
+ 
v"u log u 

1 

On the other hand, using the estilnates for Sq given in Lemlna 2.5, we obtain 

y<q<_u 

Sq 
y<_q<_u 

y<qSu 

We choose 

u = log x 
*(loglogx 91ogN) (see (6)) and that D is bounded, since E 
and recall that y = g - _ 
bas Chi. Then, from (9), (10) and (11) we get 
(12) M°(x,g, 2vfT) =0 og  • 
(logx)(log N= (loglog v= ) 

For the second terln in (8) we have 

(3) 

where 

r(x,Q(E[q])/Q) := #{p _< x: ap = 0 and p splits completely in Q( E[q]) /Q }. 
We observe that if p is a prime of supersingular reduction that splits completely in 
some Q(E[q])/Q, then q = 2. Indeed. for such primes p and q we have, on the one 
hand, that q](p + 1 - ap) = (p + 1), and, on the other hand, that q](p - 1): thus 
q[2. Now we note that in the sure of (13) we run over q > y; thus, by out choice of 
y (see (6)), q  2. This implies that 
(14) M'(x,y,2v) =0. 



2660 ALINA CARMEN COJOCARU 

#(k) | li X. 

--, t,(/) t,(k) V.,, .(k) 
» ,,(») -  ,,(.) /--- ,,(»)' 

where k" 
which there exists a prime diviser q > y. Using part '2 of Lelmna 2.1 we get that 

lneans that the Slllll is over those positive square-free integers k for 

"p(k) x 1 
lix «< log----TZZqt3/2 
• q>y t=l 

log :r log :r " 
-- ) 0og log -- ) 

.1" 

\ 
log log x ] 
log x J " 
log W- 

T}IllS 

(log x)y log B 
= O ((logx)(log 

=  log x 
(15) f(:r,Q) Iœelix +O (l°gz(l°gl°gw-) 
This completes t he proof of Theorem 1.1. 

3.4. The proof of Corollary 1.2. First, let us recall that it was pointed out bv 
Serre (see [Mu3, p. 327]) that the density fE is positive if and only if Q(E[2]) ¢- Q. 
Now, we note that a necessary condition for forinula (15) to hold is that x _> 
e.p (V). Shen, if  × e.p (V). the ,,in terre of (15) ,,-iii be bigger than the 
error terln. This proves the assertion of the corollary. 

4. CONCLUDING REMARKS 

As mentioned in the proof of Corollary 1.2, the density [E is positive if and only 
if Q(E[2]) ¢- Q. For the sake of clarity, we explain this in what follows in the case 
of a CM elliptic curve/3 defined over Q. Naturally, in order te have Ioe ¢- 0 we need 
te assume Q(E[2]) ¢ Q, for otherwise the torsion part of E(Q) contains the Klein 
four group and se E(Fv) calmer be cvclic. The condition is also suflàcient. Te see 
this, let us first note that if Q(E[2]) ¢ Q, then [Q(E[2]) -Q] is 2, 3 or 6. We let 
K be the Ulfique abelian subextension contained in Q(E[2]). Also, we let Iç be the 
CM field of E. We recall that Iç(E[q]) = Q(E[q]) for any prime q > 3 (see [Mul, 
p. 165, Lelnlna 6]), and we observe that since K is a quadratic field and Iç2 is a 
cubic or a quadratic field, we have either K ç? K = Q or Iç = K. If K VI K = Q, 



Ch CLICITY OF CM ELLIPTIC CURVES MODULO p 2661 

then using that K2 ç Q(E[2]) and Ix" ç Q(E[q]) for anv q _> 3. we deduce that the 
densitv of the primes p that do hOt split completely in anv of the fields Q(E[q]) is 
greater than or equal to the density of the prilnes p that do not split COlnpletely in 
K 2 and N. In other vords, 
( )( )» 
c   [ç.] 1 [¢.] _ . 
If N2 = N, then K ç Q(E[q]) for any prime q. and so the ,h'nsity of the primes p 
tha¢ do hot si,lit complotely in any of the fiehls Q(E[q]) is greater than or equal to 
the density of thc primes p that do hot si,lit COlnI)letely in N. In other xvords, 
( I ) , 
f > 1 [Iç Q >- 
This completes the proof of the positivity of . 
The main significance of our uncomtitional proof of the asymptotic formula for 
f(x,Q) in the case of a ('gl elliptic curve lies in thc silnpli«ity of the tools that 
are used. Rare Mm'ty's initial proof avoided the (IH bv using a di«ult appli- 
cation of the large sieve for lmmber fields, nalnely a Bombieri-Vinogradov type 
result for mmfl)er fields. In our nev proof we use instead an application of the 
sieve of Eratosthenes, one of the silnt)lest sieves in nmnber theory.  point out 
that this application of the sieve of Eratosthenes (Lemma 2.5) could be viewed as 
a Brun-Titchmarsh theorem for quadratic mmfl)er fields, since it gives nontrivial 
upper bomds for the lmllfler of (prilwipal) prilne ideals whose gelmrator satisfies 
congruente conditions. A result of this kind had been obtained in [Sch], but as an 
application of the large sieve for mmber fields, and «ould hm'e been used in our 
treatment of M(z. g. 2). 
Another significance of our new proof is that it provides explicit error terres, 
with absolute O-constants. As noted in Corollary 1.2. we can then deduce an 
unconditional upper bound for the smallest prime p for which E(Fp) is cyclic. 
Considerable improvements of this bound, mder GRH. will be discussed in an 
upconfing paper. 
Naturally, one can ask if our ideas can be explored finther and used in other 
related situations. For example, one could consider the question of determining 
the number of prime ideals for which the reduction of a CM elliptic curve defined 
over a number field gives a cvclic group. It seems that out tools can be used iii 
this situation. Another question is that of using the ideas of this paper in the case 
of a non-CM elliptic curve. At present no unconditiolml proof for the asymptotic 
formula for f(z, Q) is known in this situation, but, as mentioned in Section 1. onlv 
a proof based on a quasi-GRH assumption (see [acC1]). If we assume a variation of 
a conjecture of Lang and 'otter on the number of distinct fields Q(gp) obtained 
when p runs over primes of ordinm'v reduction for a non-CM elliptic curve (see 
[LT1]). then it turns out that we can follow the current CM approach even in 
1 
the non-C case. The dependence  on the discriminant D of the estimates 
provided by Lemma 2.5 vill be more advantageous than the dependence on D 
provided bv Schaal's result memioned above. This is. again, ail asset of our new 
proof. Yet another related question is that of determilfing an asymptotic fornmla 
for the number of primes p for which the order of E(p) is square-free. The ideas 
of our paper can be successflflly used to answer this question if E is a Cg[ elliptic 
curve. The details of our last two claires will be given in different upcoming papers. 



2662 ALINA CARNIEN COJOCARU 

ACKNOWLEDGEIvIENTS 
The results of thi paper are part of my doctoral thesis [acC2]. I express my 
deepest gratitude to my supervisor, Professor M. Rare Murty, for all his help and 
support. I ara also gratefld to Professor Erut Kani for usefld discussions on the 
algebraic prelimiuaries of the paper. 
tEFERENCES 
[acC1] A. C. Cojocaru, "On the cyclicity of the group of Fp-rational points of non-CM elliptic 
curves", Journal of Number Theory, vol. 96, no. 2, October 2002, pp. 335-350. 
[acÇ2] A. C Cojocaru. "Cyclicity of elliptic curves modulo p", Ph.D. thesis, Queen's University, 
Kingston, Cnrtda, 21|(12. 
[BlkIP] I. Borosh, C. J. Moreno, and H. Porta, "Elliptic curves over finite fields II", Mathematics 
of Computation. vol. 29, July 1975, pp. 951-964. lkIR 53:8067 
[Ho] C. Hooley, "Applications of sieve methods to the theory of numbers". Cambridge University 
Press, 1976. IvIR 53:7976 
[LT1] S. Lang and H. Trotter, "Frobenius distributions in GL2-extensions". Lecture Notes in 
Mathematics 504. Springer-Verlag, 1976. lkllq 58:27900 
[LT2] S. Lang and H. Trotter, 'Primitive points on elliptic curves", Bulletin of the American 
Mathematical Society, vol. 83, uo. 2, March 1977, pp. 289-292. MR 55:308 
[Mul] M. Rare Murty, "On Artin's conjecture", Journal of Number Theory, vol. 16, no. 2, April 
1983, pp. 147-168. M1R 86f:11087 
[Mu2] M. Raln Murty, "An analogue of Artin's conjecture for abelian extensions", Journal of 
Number Theory, vol. 18. no. 3, June 1984. pp. 241-248. lkIR 85j:11161 
[lklu3] M. Rare Murty, "Artin's conjecture and elliptic analogues", Sieve Methods, Exponential 
Sums and their Applications in Number Theory (eds. G. R. H. Greaves. G. Harman, M. 
N. Huxley), Cambridge University Press, 1996, pp. 326-344. lkIR 2000a:11098 
[Mu4] M. lRam Murty, "Problems in analytic number theory", Graduate Texts in Mathematics 
206, Springer-Verlag, 2001. lkIF[ 2001k:11002 
[Sch] W. Schaal, "On the large sieve method in algebraic number fields", Journal of Number 
Theory 2, 1970, pp. 249-270. MR 42:7626 
[Sel] J. -P. Serre, "Résumé des cours de 1977-1978", Annuaire du Collège de France 1978, pp. 
67-70. 
[Se2] J. -P. Serre, "Quelques applications du théorème de densité de Chebotarev", Inst. Hautes 
Etudes Sci. Publ. Math.. no. 54. 1981, pp. 123-201. M1R 83k:12011 
[Silvl] J. H. Silverman, "The arithmetic of elliptic curves", Graduate Texts in Mathematics 106, 
Springer-Verlag, New York, 1986. lkIR 87g:11070 
[Silv2] J. H. Silverman, "Advanced topics in the arithmetic of elliptic curves". Graduate Texts in 
Mathematics 151, Spriuger-Verlag, New York, 1994. lkIR 96b:11074 

DEPARTMENT OF lklATHENIATICS AND STAT1STICS, QUEEN'S UNIVERSITY, K1NGSTON, ONTARIO, 
CNADA, KïL 3N6 
E-mail address : alina@mast.queensu, ca 
Current address: The Fields Institute for Research iu lklathematical Sciences. 222 College 
Street, Toronto, Ontario, M5T 3J1, Canada 
E-mail address: alina@fields.utoronto, ca 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2663-2674 
S 0002-9947(03)03194-5 
Article electronically published on February 27, 2003 

TAYLOR EXPANSION OF AN EISENSTEIN SERIES 

TONGHAI YANG 

ABSTRACT. In this paper, we give an explicit formula for the first two terres 
of the Taylor expansion of a classical Eisenstein series of weight 2k ÷ 1 for 
F0(q). Both the first ternl and the secoud terre have interesting arithmetic 
interpretations. \Ve apply the result to compute the central derivative of some 
Hecke L-functions. 

0. INTRODUCTION 
Consider the classical Eisenstein series 
 Iln(7"r) s, 
which bas a simple pole at s = 1. The wcll-known Kronecker limit formula gives 
a closed formula for the next terre (the constant terln) in terlns of the Dcdckind 
rkfunction and has a lot of applications in nulnber theorv. It seems natural and 
worthwhile to studv the saine question for more general Eisenstein series. For 
example, consider the Eiscnstein series 
(0.1) E(r,s) :  e(d)(«r + d) -2k-' I,n(Tr) -k. 
-eP\Po(q) 
Here 7 = ( d b ), --q is a fundamental discriminant of an imaginary quadratic fiel& 
and e = (-q). This Eisenstein series was used in the celebrated work of Gross and 
Zagier ([GZ, Chapter IV]) to compute the central derivative of cuspidal modular 
forms of weight 2k + 2. The Eisenstein series is holomorphic (as a function of s) at 
the symlnetric center s = 0 with the leading terre (constant terre) given by a theta 
series via the Siegel-Weil formula. The almlogne of the Kronecker limit formula 
would be a closed formula for the central derivative at s = 0 -the main object 
of this paper. This would give a direct proof of [GZ, Proposition 4.5]. Another 
application is to give a closed formula for the central derivative of a family of Hccke 
L-series associated to Chi abelian varieties, which is very important in the arith- 
metic of Chi abelian varieties in view of the Birch and Swinnerton-Dver conjecture. 
This application will be given in section 4. We will also prove a transformation 
equation for the tangent line of the Eisenstein series at the center, which should be 
of independent interest. 

Received by the editors September 9, 2002. 
2000 Mathematics Subject Classification. Primary llG05, llM20, 14H52. 
Key words and phrases. Kronecker formula, central derivative, elliptic curves, Eisenstein series. 
Partially supported by an ANIS Centennial fellowship and NSF grant DMS-0070476. 

@2003 American Mathematical Societ) 



2664 TONGHAI YANG 

To make the exposition silnple, we assume that q > 3 is a prime congruent to 3 
lnodulo 4. Let k = Q( 
Set 
A(s,) = -V( s+ 1)L(s, ) 

(0.2) 
and 
(0.3) 

/*(T,) 
= q  A(s + 1, «)E(T, 
It is well known that E*(r, s) is hololnorphic. 
As in [GZ, Propositions 4.4 and 3.3], we define 

k 
(0.a) p(t) =  ()  (-t),,, 

alld 
(0.5) qt.(t) __ e-t(u- 1)ku--du, t > 0. 
\;e rena.rk that p(-t) and qt.(t) are two «basic" solutions of the differential equa- 
tions 
(0.6) ,C"() + ( + t)C'() - »C() = 0. 
Filmlly. let p(n) be given by 
(0.7) <() =  (,,)- 
Theorem 0.1. Let the notatwn be as above, and let h be the ideal class number of 
k. lI'rite r = u + iv. Then 
E*(r, 0)= v-(h + 2 __,p(n)p(4rcnv)e(nr)) 

and 

and 

ao(v) = h 

( A'(1, e)  1) 
log(qv) + 2 A(1, e + Z  

a = (ordqt + 1)p(r01og q +  (ordpr + 1)p(n/p)logp. 
(E)=--I 
The forlnulas should be colnpared to those for  in [GZ, Propositions 4.4 and 
4.5]. In fact, lnultiplying our forlnulas by the theta function in their paper and 
taking the trace would yield their formulas for ç. The method used here seelns 
to be more suitable fol" generalization. The proof is based on the observation that 
the Eisenstein series (0.1) can be split into two Eisenstein series. One of them is 
coherent, and it is easy to compute its value. It contributes little to the central 



TAYLOIR EXPANSION OF AN EISENSTEIN SERIES 2665 

derivative. The other one is incohel'ent, contril)utes nothing to the value, a.nd its 
central derivative can be computed by the method of [KRY], where we dealt with 
the case k = 0. This consists of sections 1 and 2. 
In section 3, we study how the value and derivative behave mder the Fricke 
involution "r  -1/qr and obtain the following fllnctional equation. One inter- 
esting point about the equatiou is that it basically follows from the definition of 
antomorphiç fornls (sec (3.2)). 
Theorem 0.2. The mo&dar forms E*(r, O) and E*'(r, O) satis.[y tbe.[ollowmg flmc- 

tioal eq-uatio: 
* 1 
( (-.o) 
E*'( -1 ol 

= i(v) 2"+ 
j=l 

--1 
_1+1 log q ,c?*'(T, 0) " 
3 

Finally, let p be a canonical Hecke charact«r of weight l of k (see section 4 for the 
definition). It is associated to the C elliptic çurve 4(q) studicd bv Gl'OSS ([Gro]). 
When q = 3 mod 8, S. Miller and the author proved re«entlv lhat the central 
derivative L'(l. 13) 7 0 ([MY]). Since thc central derivative encodes vcry important 
information in the arithmctic of A(q), it is iml»ortalt to film a good formula for the 
central derivative. Standm'd calculation shows that the L-series L(s. I 3) is E(r, 2s) 
evaluated at a ÇM cycle. So Theorem Il.1 gives an explicit forlmlla for the central 
derivative L(1,13) (Crollary 4.2). 

1. COHERENT AND INCOHERENT EISENSTEIN SEl:lIES 

Let G = SL2 over Q, and let B = TN be the standard Borel subgroup, where 
T is the standard maxinml split torus of B and N is the unipotent radical of B. 
Their ratiolml points are given bv 

and 

a-  a } 

Consider the global induced representation 
I(s. e) 
= IndB(A) ] ] 
of ((N), where N is the ring of adèles of Q. Bv definition a section (s)  I(s. «) 
satisfies 
(1.1) (n(b)m(a)g. s) = «(a)lal+(g, s) 
for a  A* and b  A. Let K = SL() and let K = SO(2)(). Associated to a 
standard section . which lneans that its restriction on IçIç is ildependent of s. 
one defines the Eisenstein series 
(1.2) E(g,s,) =  (2g, s). 
eB(Q)ça(Q) 
It is absolutely convergent for e s > 1 and has a lneromorphic continuation to 
the whole complex s-plane. X consider three standard sections 0, i in this 
paper. For every priine p { q, let p  I(s. ep) be the unique spherical section 



2666 TONGHAI YANG 

such that (I)p(x) = 1 for every ,r C Kp = SL2(Zp). Let (I) C I(s, e) be the unique 
section of weight 2k + 1 in the sense that 
(1.3) (gko, s) = .(9, s) e(+)0 
for everv ko ( cosO sinO 
 = -iOcoO) K" Forp=q, let 
4= cq 
be the Iwahori subgroup of Kq. Then % defines a character of .lq via 
(1.) «( cq 
As described in [KY, section 2], the subspace of I(s. Q) consisting of % eigenvec- 
i determined 
tors of Jq is two-dimensional and is spammd by the cell fimctions of q, 
by 
(1.5) 
• qw/, s) = 5 0, where w0 = 1 and u, = w = . 
ç denote this subspace bv II (,lq, eq, s). A better basis for this subspace turns out 
to be given 
1 
(1.6) çq 
which are "eigenflmctions" of somc intcrtwining operator (see Lemma 2.2). Set 
(1.7) 0 o 
= q   and = q  p. 
Clearlv, 0 1 + -). For +iv with v 
. = ( + r=u >0.1et 
(1.8) g = ,,(),(). 
Then standard computation gives 
Proposition 1.1. Let te notatio be as above. Then 
.(, )= .-},(g, , 0) 
= -,,--i(*(g,,+) + *(g ,-)). 
2 ' 

Here 

E*(g,s, ep) = q A(s + 1,e)E(g,s, dp) 
is the completion of the Eisenstein series E(g, s, (I)). 
As we will see in Proposition 2.4, the Eisensteiu series with (I) + behave almost as 
"even/odd" functions respectively, and both have nice functional equations. This 
is hot a coincidence. Indeed, ri'oto the point of view of representation theory, 
• +(g, 0) is a coherent section in I(0, e) in the sense that it cornes from a global 
(two-dimensional) quadrat.ic space, while (I)-(g, 0) is an incoherent section in I(0, e), 
coming from a collection of inconsistent local quadrat.ic spaces. We refer to [Ku] 
for explanation of this terminology and for a general idea for computing the cen- 
tral derivative of incoherent Eisenstein series. Every section in I(0, e) is a linear 
combination of coherent and incoherent sections; we just made it explicit in this 
case. 



TAt LOR EXPANSION OF AN EISENSTEIN SERIES 2667 

2. PROOF OF THEOREM 0.1 
Let 0 = I-I 0p be the "canonicar' additive character of A via 
{e 2rix if p = oe, 
(,p(x) = e_2( ) if p -¢ x. 
Here A is the canonical map Qp  Qp/Zp "- Q/Z. For a standard section 
= I-I (p E I(s, e) and d E Q, one defincs thc local Whittaker flmction 
IVa,p(9, s, ¢) =  ¢(u,.,(b)9,s)Op(-db)ab. 

(2.1) 
Let 
(2.2) 

be its completion. \Ve also set lp(s) 
M M (s) is a nornlalized intertwning opcrat or from I(s. ) to I (-s, ). 
In gencral, an Eisenstein stries E*(9, s, ç) has a Fourier ext)ansion 
(2.3) E*(g,s, 
d 

with 
(2.4) 

E(9, s,4P) = q 
P 
for d ¢ 0 and 
• -,-a «,  
(2.5) E(9, s,ç)=q  A(s+l,¢)ç(g,s)+q  5I (s)ç(g,s). 
The local Whittaker integrals are computed in the next three lcmmas. 
Lemma 2.1 ([KRY, Lemma 2.4]). For a flnite prime number p ¢ q. 
II a,p1, s, p) = 0 vless ordp d > 0. h such a case, one bas 
ord. d 
v%,(1. . %) =  (,(v)p-)  
r=O 

and 

M(s)((s) = Lp(s, ()(p(-s). 
Here çp is the unique spherical section defined in section 1. In particular 
where pp(d) -=- p(pord»d) for p < oe. 
Lemma 2.2. For p = q. one bas 

lV*,(Wl, s. '+ = 

one as 

(l+(q(d)q-S(°rdqd+l)) (±--- ) 
\ --q 
0 

if ordq d _> 0, 
if ordq d = - 1, 
otherwise 



2668 TONGHAIYANG 

Mq(s)q 1 + 
= -'--------q . 

Pro@ The tir,st tbrnmla follows flore [KIY, (3.26)-(3.29)]. For the second formula, 
notice that AI,(s) is an itertwinig operator between eigenspaces ll'(Jq, eq, s) and 
:(./. ,-) of .,. 
.  • + b 
.I, (S)q = a q + 
for sonc constants a  and b . Phgging in 9 = u'o and t,, and appl)ing the first 
formula, one gets the desired fbrnmla. 
Lema 2.3. Let • =  be the local section in I(s. e) defined b9(1.3). 

Il(g. l. t3, ) = / e-gx(._ • 31- h) a-1 (.F - h)O-d.Æ 
-Fh>0 
is Shi,tura  «la fu,cliot for 9 > O. h  N, atd e  and e  s'ucietttl9 large 
(2) For d > O. o« bas 
1 
ll.(g.0. ) = 2it,pk(4dt,)«(dr), 
where pk is deflned b (0.4). 
(3) For d < O , one has ll..(g.O, ) = O. and 
1 
1I 3. (g. o.  ) = i,, q»(-4dt,)e( dT). 
where q is 9i,,e, by (0.5). 
 j-sC 
(4) oE()() = i =0 +/2(-)e(-) • 
Pro@ The proof is the sae as that of [KRY. Proposition 2.6] and is left to the 
reader.  

Proposition 2.4. 

(2.6) 

k k 
[[(j - )F*(g,-, e ) = : H(J + )F*(r,., 

Pro@ By Lenmlas 2.1-2.3. one has 

Now the proposition follows from the fimctional eqta.t.ions 
and 
E(9, s, ) = E(g,-s,l(s)). 
Here I(s) = M*(')A(s + 1. )- is the unnormalized intertwining operator from 



TAYLOR EXPANSION OF AN EISENSTEIN SERIES 2669 

Theorem 2.5. One bas 
v (T,o., +) = 2(hq+ 2 

('.8) 
atd 
(2.9) 

1 *l -- 

Proof. First. we observe that 
(2.10) H pp(d)(1 + %(d)) 
SillCe 

= p(Idl)(1 + %(d)) = 2p(d) 

1 = H e(d)= sigl,(d)eq(d)II(-1) 

poe pi d 

Fornmla (2.8) is a special case of the Siegel-Weil forlnula. \Ve give a direct proof 
here using Lemmas 2.1-2.3. First, the lelnmas ilnply E(gT, 0. • +) = 0 Ulfless d >_ 0 
is an integer. \Vhen d > 0 is an integer, the lemlnas and (2.10) ilnply 

• - l H 
Ea(g-, O, • -) = q pp(d) 

1 + eq(d)2iv½pk(4rdv)e(dr) 

1 
= 4vp(d)pk(47dv)e(dr). 

(2.11) 
Tlle saine |elnmas also ilnply 
E(g-, 0.  +) = qA(1,¢)+(g,-.O) + q½M*(O)+(g-.O) 
1 1 
= hv + A(O, 
1 
= 2hv. 

This proves (2.8). 
As for (2.9), we again check terln by terre, and it is clear frolll tlle leIllllltlS that 
E'(g-, 0, -) = 0 uIfless d is an integer, which we assume froln now on. 
WheI d < O, II'd.,(9-,O,'b-) = 0 b3 Lemllla 2.3(3), and so (using Lemlnas 
2.1-2.3 and (2.10)) 
"*' ' 0 -* , 
e'(9»0,-)=qu.tg, . oe)t,a(l 0, Il 
= -2,,½q(-v)«()/1 - )) Il 

1 
= -2v - p(-d)qk(-47rdv)e(dr), 

as desired. 
VVhen d > 0 and eqd) = 1, one has II'*.q(1.O, -) =0 and 
-1 
IId,q(1,O, 4 -) -- v/(ordq d + 1) logq. 



2670 TONGHAI YANG 

The saine computation using Letnnlas 2.1-2.3 and (2.10) yields 
*I -- 1 
E e (g-, 0, (I)) = -2vpk(4nd'v)e(dr)(ordqd + 1)p(d)tog q 
1 
(2.12) = -2va,pk(4ndv)e(dr), 
since a = (ordq d+ 1)p(d)logq in this case. 
When d > 0 and eq(d) = -1. there is a prinle lld such that II)*t(1,0,t ) = 
pt(d) = 0 by 22.10). In this case, 
1 (ordt d + 1) log l. 
The saine calulation yelds 
E d (g,O. -) = -2va,,p(4dv)«(d7), 

as desired. 
Finally, 

(2.13) 

with 

22.14) 

go 

when d = 0, one has by the Saille lenlnlas, 
1 
t;(gT, s, +) = k . (G(s) + G(-0) 

k 
G(s) = (qv)  A(1 + s. e) H( j + -). 
j=l 

(2.15) E('(g-, 0, (I)-)- 

"G--420) = hv½ Çlog(q.')+ 2-- 

This finishes the proof of (2.9). 

A(1,e) 

[] 

One has by Proposition 2.4. 
*(,o) lv--' * 
= E (g»O,,O +) 
2 

Proof of Theorem 0.1. 
(2.16) 
and 
(2.17) E*('r, 0) = v --½ lE *' l klE* ] 
L t..o.-)- ,=7 (-'°+) " 
Now Theorem 0.1 easily follows from Propositions 1.1 and 2.4 and Theorem 2.5. 

3. PROOF OF THEOREM 0.2 

By Proposition 1.1 and Fornmlas (2.16) and (2.17), Theorem 0.2 is equivalent 
to the identity 
()2#+ ( -1 ï)(E'(9-,0.+,' 
(/*(g_,O,5+) =i l log q I k (gr.O,ê),]" 
(3.1) E'*' - 
t,s*'(-,o,<-))  
To prove (3.1), one observes the following trivial but fmidamental identitv and 
computes both sides: 
(3.2) • --1 * 
s; ( g«,,,I,-) = s; (w<,,-,,,I,+). 



TAYLOR EXPANSION OF AN EISENSTEIN SEIIES 2671 

Here wf and tt, are the images of w = (ï 1 ) in C,(Af) and G(IR) respectivelv. 
The left-hand side of this identity is given by 

Lemma 3.1. 

For the right-hand side of (3.2), Olle has 

Lemlna 3.2. 

E*'(Wfgq-,O, dP-)] =i ½1ogq E*'(g-,O,(P-)]" 

Pro@ We verify these identities by comparing the Fourier coefficients E (WfgqT, 
s, dp +) with E(9, s, dp+). We nmy assulne that d is an integer bv Lemnms 2.1-2.3. 
Straightforward calculation using the saine lenmlas yields, for any integer d, 
(3.3) IV,»(Wygq¢,S, dp +) = F(d)II'.»(gT, s, dp +) 
with 

{ ç if p { qoc, 
(3.4) F(d) = '- ifp= OE, 
_1_ 1 l+eq(d)q -st 
 l+%(d)q_(+} if p = q. 
Here r = ordq d. We will verif.v the derivative part and leave the value part to the 
reader. First assume d -fl 0. It follows rioto (3.3) that 
E*" lE*'" ! ifeq(d)=-l, 
a_ (Wfgqr, 0, (I)-) = d tg-, 0, (P-) [,i  if {[q (d) = 1. 
q ordq d+l 
When eq(d) = 1, one bas by (2.11) and (2.12), 

No 

d g-,O, dP-) _Ed(g,O,+)ordqd+ 1 logq. 

(3.5) 

= Ed (gr,O, P-) +  logqEd(gr, O, +), 
Ee_ (WfgqT, O, (--) . ,, i 
q 

as desired. When eq(d) = -1 we have E(g-, ¢0, (I)+) = 0, and (3.5) still holds. 
It remains to check the constant term. Recall (2.13)-(2.15). Direct calculation 
using Lemmas 2.1-2.3 also gives 
(3.6) E)(Wfgqr,S, dP+)=:[:  (q-G(s)+q-G(-s)). 
I]j=l (J -}- ) 



2672 TONGHAI YANG 

Therefore, 

(3.7) 

E.,, , 2G'(0) .2G(0) l log q 
0 uf9qr'O'dP-) =i k--, q-Z k! 2 
i 
= iE'(g,-, 0. qb-) +  logqE(gq-.O. b+), 

a.s expected, too. 

4. L-SERIES 
Recall that q is a prime congruent to 3 modulo 4 and k = Q(v) is the 
associated imagi,mry quadratic field. Recall also ([Rob]) that a canoifical Hecke 
character of k of weight 2/," + 1 is a Hccke character p satisfying 
(1) The conductor of p is 
(2) p('2t) = It(ç() fiw an idem 1 relativcly l)rime to 
(3) t(aOk) = +o 2k+l. 
In this section, we will give an explicit forlmfla for thc central derivative of its 
L-fllnct.ion, which has dcep arithlnetic implications as mcntioned iii the introduc- 
tion. \Ve rcfer to [Gro] tbr the arithlnetics of elliptic curves associated to these 
Hccke characters (sec also [iXlY] and [Ya] and the refcrence there for more recent 
devclopments). For each ideal class C of k, we can define the partial L-series bv 
(4.) c(.,t,, c) = 
C, integral 
Of course, L(s, #) = -ce CL(k) L(s, t. C). The following proposition is standard. 
Proposition 4.1. Let  Ç Che a primitive ideal of le relatively prime to 2q. and 
u,rite 
9 = [a, b + V/ ] with a > O. b = O modq. 
9 

b+ v' 

2aq 

L(s+l,'+ 1. p. C') -- 

(4.6) 

¢(r) = ao(V) - 2 Z a,pk(4rrm')e(nr) - 2 Z p(-,,)q,.(-4rrm,)e(r-r). 

E* (r, 0) = v-O(r) 

t,("a) 
(.oE.o.l)2k+l (2v/)-L(2s + l,e)E(r,2). 

and 
Then Theorem 0.1 savs that 
and 



TA5 LOR EXPANSION OF AN EISENSTEIN SERIES 2673 

Corollary 4.2. Let the notation be as in Proposition 4.1. 
(1) The central L-value is 

L'(k + 1, p., trivial) 

7r 1 i 

Pro@ Only the second one needs a little explalmtion. When (-1)k() = -1 ,ve have L(k + 1. p. C) = {1 automaticallv and thus O.(r,a) = ). So Theorem 0.1 and 
Proposition 4.1 imply 

L'(k + 1, p, C) = 

When C is trivial, one tan take  = Otc. In this case, a = 1 and 
and ttms 
çb(r,a)=çb(-+)._ 

b -- 1 lllod 1, 
2q -- 2 

In recent joint work with S. Miller ([MY]), xe proved that L'(1,p,trivial) > 0 
when q = 3 mod 8 and k = 0. Combining that with Corollary 4.2, om has the 
following curious inequality: 

(4.7) 



2674 TONGHAI YANG 

ACKNOWLEDGEMENT 

This work was inspired by joint work with Steve Kudla and Michael Rapoport. 
The author thanks theln for the inspiration. He thanks IRene Schoof for numerically 
verifving the formulae iii Corollary 4.2, which corrects a lnistake in an earlier version 
of this paper. Finally, |le thanks Dick Gross, Steve Miller. and David Rohrlich for 
stimulating discussions. 

EFERENCES 

[Gro] 
[KI:IY l 
[lohl 
[Shl 
[Y] 

B. Gross, Arithmetic on elliptic curves with complex multiplication, Lecture Notes in 
Math., no. 776, Springer-Verlag, Berlin, 1980. MR 81f:10041 
B. Gross and D. Zagier, Heegner points and derivatives of L-series, Invent. Math. 84 
(1986), 225-320. Mit 87j.-11057 
S. Kudla, Central derivatives of Eisenstein semes and height pairings, Ann. Math. 146 
(1997), 545-646. MR 99j: 11047 
S. Kudla, M. Rapoport, and T.H. Yang, On the demvative of an Eisenstein series of 
weight one, Internat. Math. I/es. Nolices 7 (1999), 347-385. MR 2000b:11057 
S. D. Millet and T. H. Yang, Non-vanishing of the central derivative of canonical Hecke 
L-functions, Math. Res. Letters 7 (2000), 263-277. MR 2001i:11058 
D. IRohrlich, Roof numbers of Hecke L-functions of CM fields, Amer. J. Malh. 104 
(1982), 517-543. MR 83j:12011 
G. Shimura, Confluent hpergeometric functions on tube domains, Math. Ann. 260 
(1982), 269-302. MR 84f:32040 
T.H. Yang, On CM abelian varieties over imaginary quadratic fields, preprint. 

DEPARTMENT OF klATHEMATICS, UNIVERSITY OF VISCONSIN. kI.\DISON. VISCONSlN 53706 
E-mail address: thyang@math.uisc, edu 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2675-2713 
S 0002-9947(0303121-0 
Article electronically published on Match 17, 2003 

SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 

ERIC FREEMAN 

ABSTRACT. ,Ve treat systems of real diagonal forms Fl(X), F(x),..., FR(x) 
of degree k, in s variables. Ve give a Iower bound so(R, k), which depends only 
on R and k, such that if s >_ s0(R. k) holds, then, under certain conditions on 
the forms, and for any positive rem number e, there is a nonzero integral sinul- 
taneous solution x E Z s of the system of Diophantine inequalities IFi(x)l <  
for 1 < i < R. In particular, our result is one of the first to treat systems 
of inequalities of even degree. The result is an extension of earlier work by 
the author on quadratic forms. Also, a restriction in that work is removed. 
which enables us to now treat combined systems of Diophantine equations and 
inequatities. 

1. INTRODUCTION 
1.1. Statement of main result. In 190. Schmidt [17] proved a far-reaching 
result about systems of Diophantine inequalities of odd degree. Given anv odd 
positive integers dl,...,d, Schmidt showed that there exists a positive integer 
si = si(d1 .... ,d), depending onlv on dl,...,d, with the following property: 
given any positive integer s _> s and any rem forms, or homogeneous polynomials, 
Gl(X) ..... G(x), in s variables, of respective degrees dl, .... d, and given any 
positive nmnber , there alwavs exists a nonzero integral vector y  Z s satisfying 
the systenl 
(1.1) IGI(y)I < e, IG(y)I < e .... , IG(y)I < e. 
So, in other words, as long as the forms are ail of odd degree, and are defined 
in enough variables in terres only of the degrees, then there is a nonzero integral 
solution of the inequalities (1.1). Many particular classes of svstems of the type 
(1.1) bave been studied. 
For Diophantine inequalities of even degree, the situation is nmch different. 
There is no such general result as above for integral solutions of Diophantine in- 
equalities of even degree, and in fact there are few results at ail for inequalities of 
even degree. (However, results are known if one allows solutions in algebraic inte- 
gers in purely imaginary number fields. See Theorem 11.1 of [22].) In this article, 
we present one of the first results concerning systems of Diophantine inequalities of 
even degree, while at the saine rime removing a restriction from an earlier paper by 
the author, on quadratic Diophantine inequalities [10]. We are now able to remove 

Received by the editors October 15, 2001. 
2000 Mathematics Subject Classification. Primary 11D75; Secondary llD41, 11D72, 11P55. 
Key words and phrases. Combined systems of Diophantine equations and inequalities, forms 
in many variables, applications of the Hardy-Littlewood method. 
The author was supported by an NSF Postdoctoral Fellowship. 

(2003 American MathematicM Societv 



'2-676 ERIC FREEMAN 

mo (/, &, ) = / 
and 
(1.5) 

the l'estrict.ion bv com|)ilfilig the powerful ideas of B(mtkus and G6tze [3] with the 
techniques of Nadesalingaln and Pitlnan [16], and bv adapting our previous work 
in [10] and [11] to treat the miuor arcs properly. 
To state out first result, we require SOlne notation and definitions. X shall be 
working with systems of diagonal forlns Fi(x) given by 
(1.2) Fi(x) 
For systems of forms Fi as in (1.2), we define the coefficient matrix of the system 
F to be the matrix 
(1.3) A = (AO)<i<n. 
ljs 
For 1  j  s, we denotc the jth çohnnn of A by Aj. 
Now suppose that ,1 is a subset of the set of indices {1,2 .... 
to I)c the submatrix of A consisting of the cohmms Ni with j  J, and we define 
r(A.) to 1)e the tank of the matrix Aj. Finally, if x  N  satisfies Fi(x) = 0 for 
1 < i < R and the matrix 
lSjSs 
is of full tank, then we sav that x is a nonsingular solution of the system F. 
Now, for integers R and k and any rem nulnber u. we define the flmctions 
(.4) 
rein (4R  + 4R + 1,384 log 16R + 5) if k = 2 
Rk log 2k 
lg2 + uklog(Rlog2k) if k is odd and k  3 
h [48k  log 3Rk 2] if k  3. 

{ ( 5 ) 
no(k,u) = nfin 2k-t - 1, k(logh+loglogk+2)+ ukloglogk 
log k 
V'e can now state out first result. 

if k = 2 
if k>3. 

Theoreln 1.1. Suppose that k is an inte9er with k >_ 2 and that R is a positive 
integer. There are absolute real positive constants C and C2 for which the following 
property holds: 
Suppose that g is an integer satisfying 
(1.6) « lllgX(?/,0 (/?,',dl) ,N0 (]x'.d2)) . 
Suppose that s is an integer satisfyin9 s >_ gR. For 1 < i < R. suppose that Fi(x) 
is a real diagonal form of de9ree k, as in (1.2). Let A be the coefficient matrix of 
the system F, as in (1.3). Assume that the followin 9 two conditions are satisfied: 
(i) Either k is odd, or there exists a real nonsin9ular solution y of the system 
FI(y) = F2(y) ----- .... F/¢(y) ----- 0. 
(ii) For every subset J C_ {1,2 ..... s}. one hs lJl <_s-e(R-r(Aj)). 
Fix any positive real number e. Then there is a nonzero inte9ral solution x G Z  of 
the system 
(1.7) IF(x)l <  for 1 < i < R. 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2677 

For general even/% Theoreln 1.1 is one of the first results of its kind to our knowl- 
edge. We note that at least somc conditions similar to Ci) and (ii) are necessary, as 
may be seen by considering the examples given after Theorem 2 of [7]. 
For odd/,', we note that Theorem 1.1 is hot very much of an improvement be.vond 
that given bv Nadesalingam and Pitman [16], and could presumably be obtailmd 
by combining their methods with results of Vaughan [19], [20] a.nd work of Woolev 
[23]. We also observe that, following the method of Section 7.2 of [16], we could 
relnove condition (ii) for odd/," if we chose to do so. 
As well, our lnethod of proof shows that mlder the conditions of Theorem 1.1, ve 
can give a lower 1)ound of the expected order of magnitude, P-, for the nmnber 
of solutions of (1.7) in a box of size P. ff»r all sufficiently large P. This was hot 
previously knowm even in the special case of systelns of inequalities of odd degree. 
We emphasize that when using our methods, condition (ii) is necessary to obtain 
this lower bound. Presulnably, one could also give an as3mptotic formula for the 
nulnber of solutions, bv combining with the methods of [12]. 
Note that we bave ex«luded the case/," = 1 from the statenlent of the theorem. In 
this case, oto" knowledgc is ranch l»etter. For h = 1. if one has 8 >_ /ï' -1- 1, a. nonzero 
solution of (1.7) may bc follnd using a l»ox principle, whcther or hot condition (ii) 
holds. (Sec the Lemma in [4].) For k = 1. one can also find many solutions of (1.7) 
in a box of size P. (Sec Lemma 1 of [9].) 

1.2. Combined systems of Diophantine equatiolls and inequalities. \Ve 
note now that it would actuallv bc faMv routine to give at lcast ont result on 
inequalities of even degrec; one could simply generalize the work in [10] by con> 
bining those techniques with the methods in [11]. However. such a generalization 
would exclude many important classes of svstems of inequalities, fol" example those 
in which some of the forms are integral. We were forced to exclude such systems 
iii [10] because of the methods we used. In out current work, we are able to treat 
these formerly excluded systems. We give some background to more full,v expla.in. 
In [10], we considered simultaneous systems of diagonal quadratic Diophantine 
inequalities. Fol" a positive integer R. definc, for 1 _< i _< R, the real quadratic 
forms 
Q(x) = a,î + a2«g +... + a,«. 
It was proved in [10] that for every positive real lmmber e, under certain conditions 
on the system of forms Q, Q_ ..... Qn, there is an integral vector x  Z  \ {0} such 
that one has 
In that paper, one of the conditions we assumed was the following. (Sec condition 
(iii) of Theorem 3 of [10].) 

(1.8) 

For each choice of (fit,/32 ..... 3n) C liRn \ {0}, there is at least 
one coefficient of fltQt + f12Q2 + ... +/3nQn that is irrational. 

This condition allowed us to use a modification of the remarkable work of Bentkus 
and G6tze [3], but excludes certain important systems ff'oto consideration. The 
restriction (1.8) rules out systelns in which one or lnore of the forms is an integral 
form, and also any system iii which any nontrivial linear combination of the forlns 



2678 ERIC FREEMAN 

is a.n integral form. The centra.l new contribution of this paper is in removing the 
condition (1.8). 
We now state a lnore technical and lnore genera.l version of Theorem 1.1. We 
require lnore nota.tion. For a. rem vector  = (/31,...,/3n) E N u and a system G of 
forms G1 (x), G2(x) ..... Gn(x), we define the forin 
(t-G)(x) =/C (x) + 2C(x) + ... + 
Also, for rem nulnbers z, wc define 
e() = e2. 
Theorem 1.2. Suppose that k is an integer with k >_ 2 and that r and R are 
integers with R >_ 1 and 0 <_ r <_ R. Then there are absolute positive real constants 
C and C with the following propertg: 
Deflne no (k, u) as in (1.5). Suppose that f is an integer satisfying 
(1.9) « >_ ,,0 (k,02) . 
Let s be an integer witb s >_ fR. Ah'o suppose, for 1 < i < H. tbat 
(x) = a,l.-î + a: +... + 
is a diagonal form witb real coefficients. Let ,4 be the coefficient matmx of the 
system F, as in (1.3). 
Assume that the following four conditions are satzsfled: 
(i) Either k is odd. or tlere exists a real nonsingular solution y of the system 
Fl(y) = F(y) ..... Fn(y) = 0. 
(ii) For every subset J ç {1.2 .... ,s}. ov bas [J[ _< s-f(R- r(Ar)). 
(iii) Tbe forms F, F ..... Ff bave integer coefficients: also. if  = (a. aœee .... , 
oR)  IR n and o • F is a rational form, tben ar+ = ct+2 ..... an = 0. 
(iv) If r >_ 1 bolds, tben there is a positive real constant c(F) suct that one has 

q=l a:(a ..... ar,q)=l 3=1 oe:l i=1 
l <_ai <_q (l_<i_<r) 

Fix any positive real number e. Then there s a nonzero integral solution x  Z  of 
the system 
Fi(y) = 0 for l<i<r, 
(1.10) 
IFi(y)l < e for r+l < i < R. 
Moreover, if we define m0 (r. k, u) as in (1.4) avd we assume that tbe condition 
e  II 0 (r, k, d 1) holds, then we may omit condition 
from 
OUF 
assumptions. 
Some discussion o( the condition (iii) is warranted here, since it is the most 
important distinction between Theorems 1.1 and 1.2. In Theorem 1.1, we consider 
systems of inequalities (1.7). Now, if one chooses e < 1. and F(x). sa.',', is actually 
an integral form, then the system (1.7) reduces to the system 
t:(y) = 0. 
IFi(y)l<e for 2<i<R. 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2679 

So, in this case, the systeln of R inequalities aetually reduces to a systeln of one 
equation and R- 1 inequalities. A similar reduction occurs if some nontrivial real 
linear Colnbilation of the forlns ri, soEv ctF1 + c'.)F2 + ... + aRFR, is an integral 
fornl. In these situations, one might say that there is açtually ail equation hidden 
iii the system of inequalities. The conditi(m (iii) ensures that there are actually r 
equations in the systeln and R - r "truc" inequalities. It is helpflll to ensure that 
there are hOt any lnore "hidden" equations becmlse it turns out that one requires 
more variables to treat the system if there are more equations present. One lna.v 
think of Theorem 1.2, more or less, as the sub-case of Theorem 1.1 in which there 
are exactly r equations present in a systeln of /? inequalities. \Ve note that the 
second clause of condition (iii) is vacuous if r = R holds. 
One might question if, in condition (iii), the terln  -F could be replaced by 
the terre aï+Fï+ + c+2F+2 + ... + cRFR, to give a slightly weaker condition 
asserting that r equations are "'hiddelf" in the systeln. It turns out that one tan 
not, as ma.v be seen by considering the example 

F(x) = 
F(x) = 

k 
"]['8 ' 

where a 3, Ci4,..., as are anv integers. Here, for r = 1 a.nd R = 2, COl(lition (iii) does 
hOt hold, since (1/x/) (F2 - F1) is an integral forln, filial thlls tac systenl (1.10) is 
equivalent iii this case, for Slnall e, to the s.vsteln 

1 
Fl(x) =  (F:(x) - Fl(X)) = 0; 

on the other hand, for any nonzero real nulnber a2, the forln ct2F 2 is not a rational 
forlnsincetheratio ((3+v)/(2+v/-))isirrational:sothissystelndoesnot 
satisfy the suggested replacement condition. So the putative replacelnent condition 
is hOt strong enough. 
We now discuss condition (iv). The tenn  is the so-called singular sertes, 
and condition (iv) silnply states that it is bounded below by a positive constant, 
a necessary precondition when using the Hardy-Littlewood method. As the last 
sentence of Theoreln 1.2 states, we could have omitted the condition from our 
assumptions iii favor of a lower bound for /. However, since the consideration of 
the singular sertes is not our central lotus in this work, we have chosen to include 
condition (iv) so that out result can be improved immediately and transparently 
if improvements arise concerning the singular sertes and the p-adic problem. This 
should certainly be possible in the case k = 2, for example. Also, we wish to clearly 
indicate that tUe condition /? _> m0 (r,k,C) is needed Olfly because of tUe p-adic 
problem. 

1.3. Related results. çVe now compare out work with other results. For even k, 
Theorem 1.1 is an analogue of a result of Davenport and Lewis, coucerning systelns 
of Diophantine equations of even degree. (See Theoreln 2 of [7].) They assmne that 



2680 ERIC FREEMAN 

a svstem of diagonal equations 
FI(x) = F2(x) ..... FR(x) = 0. 
of even degree k. with k > 2, has a real nonsingular solution and also that foi" 
1 _< S _< R, every set of S independent integral linear combinations of Fa ..... FR 
contains at least 
(1.11) [4RSk 2 log (3Rk2)] 
variables that aI)p«'ar explicitly. Under these conditions, the system of equations 
bas a nonzero integral solution. \Ve note that if one replaces the quantity in (1.11) 
bv Sé, and restricts to integral forms, then one can show that their second condition 
is equivalent to condition (ii) of Theorem 1,1. 
Nadesalingam and Pitman [16] proved that any R real diagonal Diophantine 
inequalities of odd degree k, with k _> 13, iii .s variables with 
. > 3Rk 2 log (3Rk) 
bave a nonzero solution. We note that thev do hot require anv condition that is 
sinfilar to condition (il) of Theowm 1.1. Also, we observe that they could certainly 
have used their mothods to obtain similar results, although with a different lower 
bomd foi" s, in the cases k < 13, but iii order to streamline the presentation thev 
did hot do so. 
Finally, we note that Briideln and Cook [5] bave given a lesult on systems of 
diagonal Diophantine inequalities of odd degree. Under certain conditions on the 
coefficient matrix of the system, they show that there is a nonzero solution of 
the system of inequalities. They require an assmnption similar to condition (ii) 
of Theorem 1.1 and also a condition that is stronger than (1.8). The lmmber of 
variables they require is on the order of Rn0 (k. C2). We also note that 
they 
call 
find a lower bomd of the expected order of inagnitude foi" the mnnber of solutions 
of their system iii a box of size t » for a sequence of positive P tending to infinity. 
although hot for all large P. as out treatment provides. 

1.4. Methods used. The general strategy of the proof is to combine the method 
of I3entkus and G6tze [3], which is very effective for Diophantine inequalities, with 
the techniques that Nadesalingam and Pitman [16] use to treat combined svs- 
teins of Diophantine equations and inequalities. We remark that the techniques of 
Nadesalingam and Pitman are themselves a combination of the Hardy-Littlewood 
method and the Davenport-Heilbronn method. Using the techniques of Nade- 
salingam and Pitmm allows us to treat those systems of inequalities that contain 
'qliddeff" equations. Fol" those who are fmniliar with their argmnent, we note that 
we do hot have a so-called residual set in our proof, as in their paper. 
One other crucial result is needed, and this involves showing that, on the minor 
arcs, oui" exponential sulns are smaller than the trivial bound. Iii previous work 
on these t,vpes of problems, including [10], [111, and essentially also [3], such a 
result was achieved by splitting the minor arcs into two regions and handling each 
separately. In this paper, we handle both of these regions together, which is hot 
onlv cleaner, but also seems to be necessary here. 
I would like to thank Scott Parsell foi" showing me how to improve Lemma 6.1. 
I would also like to thank Michael Knapp and Professor Woolev for indicating to 
me how to prove part of Lemma 8.5. 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2681 

'2. DEDIÇTION OF THEOIEM 1.1 FFIOlkl THEOIEM 1.2 
The lmlk of this pal»er is dedicated te proving Theoreln 1.2. In this section. 
however, we delnonstrate how Theoreln 1.2 implies Theoreln 1.1. Te this end, we 
consider a system le of real diagonal ferres F1, Fa,..., FR as in Theorenl 1.1. 
We give a definition first. Suppose tlTat G1, G9 ..... GR and lit, ici.,_ ..... li/ are 
two svstenls of forlns. If there exists a set of R lilmarly independent real vectors 
/3, fie ..... /3R E IR R such that 
H(x)=/3-G fir 1 <i< R, 
then we say that the svstem H is equivalent te the system G, whi«h we denote by 
G  H. It is easv te check thaI this is in fa«t an equivalence relation. \Ve olserve 
as well that if G is, in particular, a system of diagonal fol'mS, and G , H holds. 
then H is also a system of diagonal ferres. 
For anv system of ferres G. we define z(G) te be the lmml)er of ferres among 
GI,G_ ..... Gn that are integral, that is, whose coeffi«ients are all integers. New 
for out systent of ferres F, we define 
r = r(F) = ma.x 
In othcr words, r(F) is shllply the lllaXillllllll llllllll)er of forlllS that are integral in 
allV systenl G equivalent te F. \Ve clearlv have 0 _< r _< B. 
New suppose that G is a system equivalent te F and that G has r integral ferres. 
Se there exist H real linearly indcpendent vectors/3./3e,...,/3/ E IR R such that 
Gi =/3, - F for 1 _< i _< R, and also the systeln G contains r integral ferres. Bv 
relabeling if necessary, we mav aSSllllle that G1, G2 ..... Gr are integral forlllS. V,e 
new show that conditions (i)-{iv) of Theorenl 1.2 hold for this systenl Cf. Then 
we will apply Theorenl 1.'2 te Cf. and we will see that the nonzero sohttion of the 
systenl G is also, under certain Coliditions, a solution of the svstelll F. 
Silice F is equivalent te Cf. if thc coefficient lnatrix of F is .4. then the coeflï«ient 
matrix of Cl is TA for the nonsingular R × R ulatrix T with rows/3,. Thus. for any 
subset J C_ { 1, 2 .... , s}, we have 
(2.1) r((TA)s) = r (TA.,) = r(As). 

The systenl F has a nonsingular sohltion 
{1.2 .... ,s} v«ith IJI = H satisfying r(Aa) 

x if and onlv if there is a sul)set ,1 of 
R and H md -j: 0. Thus the existence 
jd 

of a real nOlsingtllar solution for G follows ff'oto the existence of such a sohltion for 
F. Se condition (i) holds for G. By (2.1), it is easy te see that condition (ii) holds 
for the coeflïicient lnatrix of G. becallse it holds fol the coeflïicient lnatrix of F. 
New we turn te showing that condition (iii) of Theorem 1.2 holds for the system 
' .... a)  IR R is a real ve«tor such that 
G. Te this end, suppose that et' = (a, au 
! i 
et'- G is a rational fonn. We need te show that o,.+1 = Or+ 2 .... = O R = 0 holds. 
This holds vacuouslv if r = R. For r < R. clearing denonlinators, we see that there 
is a nonzero integer n such that. defining et = et', we have 
et-G = .et'-G  Z[x]. 
Since  is llOllzero, to prove that a+ 1¢ = a+ 2¢ .... - = 0 Re = 0 holds, it is enough 
to prove that we have c+l = Ctr+2 ..... OR = 0. 
So SUl)pose that this is hot the case. Lett.ing el, e ..... eR 1)e the standard relit 
basis for IR R, we then have that e.e2 ..... er. et are r+ 1 lilmarly ilMependent 



2682 ERIC FREEMAN 

vectors in R. çVe inay thus extend this set to a basis, sa.,," 9',9', .... 3' of 
N n, with i = ci for 1  i  r, and r+l = " Then the system of forms 
1 " G, 2 " G .... ,n - G is equiwdent to G, and thus in turn to F. But its first 
r + 1 forms are integral. This contradicts the definition of r(F), and thus we lnust 
in fact have 
-r+l  r+2  "'"  R  0. 
Thus condition (iii) holds for G. 
since  satisfies (1.6), wP havÇ g oE m0 (,k, Çl). alld tlms we certainlv 
Now, 
have g  mo (r, k, Ç). whence by the final sentence of Theorem 1.2, condition (iv) 
is UlmeCeSSaly. Since we have also secn that conditions (i)-(iii) of Theorem 1.2 hold. 
we lnay apply Theoreln 1.2 to the system G. 
Before doing so, we give some more notation. For a vector x = (z, w ..... x)  
N , define 

Ixl = ,l,aX Ixl. 
l<j<s 
For an R x s matrix BI = (tlij)l<i<R, We define 

l<j<s 
We note that the notation differs slightly from that used b5 some other authors, 
for exalnple Nadesalingam and Pitlnan [16]. Similarly, for a system F as in (1.2), 
we define 

IIFII = II/'XR Iijl • 

Defining the matrix T as above, we certainly have det(T) ¢ 0 and I[Tll ¢ 0: 
so we may apply Theorem 1.2 to the system G with e replaced by the quantity 
(Idet(T)l¢) / (RIITII-). X obtain a nonzero integral solution x e Z s of the 
system 
Idet(T)l 
la(x)l < (1 < i < ). 
IITII -1 
By Çramer's rule and Hadalnard's rule, iI follows that x is also a solution of the 
systeln (1.7), whence Theoreln 1.1 follows. 
çç now turn to the proof of Theoreln 1.2, which colnprises the rest of the paper. 

3. INITIAL ]EDUCTIONS 

In this section, we reduce the problem of proving Theorem 1.2 to the consider- 
ation of a systeln of forms as in Theorem 1.2. but under a few more restrictions. 
This will lnake our application of the Hardy-Littlewood method casier. We first 
note that by considering the forms e-lFi, it is enough to consider only the case 

çVe can also assume that we have 

IIFII > 1. 
If this were not the case, then (1.0.0 ..... O) would be a solution of the system 
(1.10) and we would be done. 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2683 

We now quote a lenmia, which seems to have been first used in this field bv Low, 
Pitmau and Wolff. (Sec Lemma 1 of [13].) It is actually a special case of a result 
on matroids, apparently due originally to Edmonds [8]. A proof tan also be fotmd 
in Aigner. (Sec Proposition 6.45 of [1].) 
Lelnlna 3.1. Let A be an Rxs "matrix over a field K and let w be a positive iuteger. 
The matrix A bas a R x Rt, pa, rtitiomble s'ubmatriz (that is. A icludes w disjoint 
Rx R submatrzces that are to,sitgula, r over K ) if a,td o'nlg if the followin 9 co,tditiot 
is satisfied: 
(3.1) [J[  s - u,(R- r(Aj)) for all sub.s'ets J ç {1.'2 ...... }. 

To be clear, by including w disjoint R × R nonsingular suhmatrices, we mean 
that there is solne pernmtation of the cohmms so that the first R cohmms form a 
nonsingular matrix, as do the second R colunms, and so on. 
Note that the condition (3.1) is exactlv condition (il) of Theorem 1.2 in thc case 
« = (. Thus we may apply the lcmma to Ihe coefficient matrix A of the svstem F, 
with the choice u' = (. Thercfore, A has ill R X ( partitionable sulmiatrix. By 
relabeling vmial)les if ne«ssary, we lnay write 
(3.2) A=[A A .... Ce Acn+ A«n+2 ... A, ], 
where A. is an R x R submatrix for 1 N v N g and where 
(a.a) A.=dt(A,,)l¢0 for lvSC. 
Now considcr the system F in the case that k is odd.  show that F has a real 
nonsingular sohltion. Since A has the form (3.2), and (3.3) holds, one can sec that 
F is equivalent to a system G with coefficient matrix B such that the left-hand 
R x 2R sulmmtrix of B has the form 
here I is the R x R identity matrix, and B.2 is a nonsingular R x R matrix.  can 
find real nunlbers z+a, z+2,..., z2 satis-ing 
zR+ [ - 1 
ZR+ 2  -1 
2 = . 
z2 -1 
Nowlet zj = 1 for 1  j  R, and let zj = 0 forj > 2R. Then for 1  j  s, 
define z = zj , which is always real, since k is odd. Setting x = (Zl,Z2 ..... 
one can observe that Gi(x) = 0 for 1 < i < R. Now the left-hand R x R matrix of 
-(OG has detenninant " -1 -1 -1 which is nonzero. Thus the 
h"   " " " Y 
system G has a real nonsingular solution, whence, as in Section 2. the svstem F 
does as well. 
Thus, whether k is odd or even, we know that there is a nonsingular solution of 
the system 
F(x)=0 (1 <i<). 
 now show that there is a real nonsingular solution whose conlponents are all 
positive. As noted above, there is a subset J = {j,j2 ..... j} Ç {1, 2 ..... s} with 



2684 EIRIC FIREEMAN 

]J] = R and a real vector x E ]s suçh that we have det(Aj) ¢ 0 and H xj ¢ O. 
Noxv, tbr 1 < i < R, we define the linear fornl 
(3.4) Li(y) = Li(y,Y2 .... ,Ys) =  AijYj. 
j:l 
• » for 1 < j < s, we see that th('re is a rem vector z = (z ..... z)  
O11 setting zj = ai _ _ 
R  such that zj ¢ 11 holds fl»rj  .1, and we have 
(3.5) L.i(z)=0 for 1 <i<R. 
Nov, if k is even. oto choice of z ensures that we have zj  0 for 1  j  s. If 
k is odd, thon for each j, we lllay if necessary replace zj by -zj, and replace the 
('oeEicients ij by --/kij for 1  i  , and consider the resulting system, hl this 
lllilllllel', Wç IIIHV ellSllre that we bave a solution z of (3.5) with zj  (I for 1  j  s 
and zj > (I for j E ,I. Note that Colditions (il) and (iii) of Theoreln 1.2 and 
«onditims (3.2) and (3.3) are mmffe«tcd. Con(litiol (iv) is also unaffected, since 
q ("  tl'l'k  zjfl.) 
the Slllll ,Ç is always rea.1 for odd k with A O  E, which mav be 
J'=[ fil 
scen lv subsfimhg -« for m h tire SUln. 
Now suppose tha z = 0 for some j0 safisfying 1  j0  s. %%% clearlv bave 
fix a positive rem mmlwr ? wih 
nllll  
Shce Ad is nonsingular, thee is  rem vtor w = (w,..., teR) such that we lmve 
A.w = -Q Aço. 
Bv Crmner's rule md Hadmnmd's rule, we certinlv hve 
I'1 -< Id«,(A,)l <-  m. 1 o t <  < . 

Now defilm 

,  zj + tt'i 

forj =ji E J 
for j = jo 
fol" j ¢ J LA {jo }. 

Writing z' ( z'. "' 
= -2 ..... Z), we have 
Li(z t) = Li(z) =0 fol" 1 <i < . 
Also, we hae ' 
 > 0 for j  JU{j0}. All ofthe other COllpOlWnts ofe' are 
equal to the respective COlnpOlmnts of e: so we bave replaced our rem nonsingular 
solution by a rem nonsingular solution that bas Olm more positive COlnpolmnt and 
tlmt still satisfios tlm çondition zj > 0 for j  3. poating this proçss as manv as 
(s - R) rimes, we can find a lonsingular real solution e = (z ..... z) with z > 0 
for ljs. 
Thus, scaling if necessal% ve lnaV choose a l'eal lmlnber  and rem munbers 
z, ï2 ..... c, tbat satisfv 
t for 1 <j<s, and 
0<Nz5 _ _ 
(.) 
L(e)=0 for l<i<R. 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2685 

To sure up, iii this section, we have delnonstrated that to prove Theorem 1.2, it 
is enough to consider a system of forms F1, F2,..., F/ as iii Theorem 1.2, with the 
added assumptions that IIFII > 1 hohts, that thc coefficient matrix .4 of the system 
satisfies the conditions (3.2) and (3.3), and that there is a real ve('tor z satisf,ving 
(3.6). Iii seçtions 4 9, we l)rove Theoreni 1.2 mlder these ad(titional assmnl)tions. 

4. THE DAVENPOIRT-HEILBRONN IIETtlOD" TttE ,¢";ETI:P 
Now we proceed with the proof of Theorenl 1.2 under the additional assmnptions 
we ruade a.bove. ' shall essentially use the Hardy-Littlewood method, in an 
involved form. X> combilie the methods of Bentkus and G6tze [3] with those of 
Nadesalingam and Pitlnan [16]. 
 note that thl'oughout the pal»er, implicil constants in the notation o() and 
O() and << a.nd >> lllay dot)end on R. s, k, , t, the coecients of the tllllS F1 ..... 
F, and the real ve«tor z. 
> cousider the lmmber of solutions of the svstem 
(4.i) F,.(y) = 0 f,,r 1 < i < ,', 
IF(y)l 
In the usual fashion, we use a real-valued, even kerncl flm('tion K • N + N to give 
a lower bomM for the nmnber of integral solutions of the svsWm {4.1) in a certain 
range. Define such a flm«tion K, fi)r any real mlmber a,, bv 
., 
By Lemma 14.1 of [2], fi)r a.ny real nmnber u, the function K satisfies the identity 
] { ,, ,f ,,, 
(4.3) ¢(u) = «(,,)K(/)S = I -I"1 if I1 < 1. 
The function K satisfies, for real nmnbel'S , the bomd 
(4.4) 
% will a.lso use the identitv 
t {1 i,,: 
(4.5) (')'« = 0 if ,, e z  {0}. 
Now for positive real numbers P and Q satisfying Q  P, we define the so-called 
Q-smooth lmmbers tobe the set 
A(P, Q) = {x 
Fix a positive real number q, to be chosen later, so that it will satisfy the require- 
ments of Lemmas 5.5 and 6.1. Then for real numbers a and P with P > 1, we 
define the exponential sure g{a) over the smooth immbers bv 
(4.6) g(a)=g(a,P)=  e (a.r). 
x(P, Pv 
Also, for 1  j  s, and real veçtors a G , we define the lillear forlns 
(4.7) 
i=1 



2686 ERIC FREEMAN 

Then we also define, for o G I t and rem nmnbers P with P _> 1, aud for 1 _< j _< s, 

the flnctious 

gj(o 0 = gj(o, P) = g (Aj(o), P) . 

(o.s) 
We define as well 
n" = [0, 11  x R-. 
Now let .lU(P) be the number of solutions of the systenl (4.1) with xj E  (P, P) 
for 1  j  s. By using the property (4.3) of the fimction K(a) and the identitv 
(4.5), one can sec that we have 
xj A(P,P n) i:l i=r+l 
ljs 
observe that this last renmrk is justified by the fact that the integral converges 
absolutcly, which follows froll (4.4), whence we mav write the integral  a product 
of R integrals. By pulling the sums into the integral, we m" rewrite the above 
bound in the form 
s R 
(4.9) 
I" j=l i=r+l 
Thus, to prove Theorem 1.2. it is enough to show that the right-hand side of (4.9) 
is at least 2. 
To this end, we give a dissection of the region of integration Il" into three subsets. 
oughly speakiug, we expect that the main contribution to the integral in (4.9) 
cornes from the region where the first r colnponeltS of  are "close" to rationM 
lmmbers with slnall denolnilmtors and the lt R - r components of  are very 
small in absolute value. We will shov that the contribution to the integral in (4.9) 
from this region, the so-called major arcs, is positive and "large", and we will also 
show that the contribution to the integral froln the other regions is slnaller, and 
thus the integral over all of Il" is positive. 
For notational ease, we set 
1 
(4.m)  = 
4(n + 1)" 
We now define, for positive integers q and integral vectors a = (al, a2 ..... af) G 
Z r, and real nmnbers P with P  2, the region ,M(q, a), or (q. a. P), bv 
(q,a) = {G[0,1]  x [-(logp)Bp-k,(logp)Bp-k]-. 
i  (logp)Bp - for 1 < i < r • 
q 
(4.11) 
here I]xl[ denotes the distance from the real number x to the nearest integer. X 
define the nmjor arcs to be the region 
(«.12) =(P)=   (q,,p), 
lq(log p)B a (mod q) 
(a ..... a ,q)=l 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2687 

where by a (mod q) we mean that a nms over vectors a E Z r such that one has 
1 aiq for 1 < i<r. 
In section 5, we will prove the existence of a functiol T(P), with T(P) k 1 and 
limp T(P) = oe, and satisfyiug a certain property. The flmction will depend on 
B and the coefficients of the forms Fi. We define the minor arcs to be the region 
(4.13) m = re(P) = ([0,1]  x [-T(P),T(P)I -) N (P). 
Finally, we define the trivial arcs to be the set 
(4.14) t= t(P) = {a e  I1 > T(P)}. 

5. AN ANALOGUE OF I, VEYL'S INEQUALITY 
In this section, we give an mmlogue of Weyl's ineqnality. For any rem number T 
with T _> 1, define the region 
(5.1) mE=roT(P)= ([0.11 r X [-T.T] R-r) \.Ad(P). 
We now state the central lenmm of this section. 

Lemma 5.1. Fix a positive real number T with T > 1. Define the forms Fi(x) as 
in (1.2) for 1 < i < B. the regio, roT(P) as above, and gj(. P) for 1  j  s as in 
(4.8). Suppose that the coeciet matrix A associated with the system F bas rank 
R. Suppose also that the irrationality coditio (iii) of Theorem 1.2 holds. Then 
one bas 
(5.2) lira sup H= 'gj(.P)' = O. 
Poe «mT(P) ps 

Observe that trivially one has H IgJ(a" P)] <- P" so we are only seeking a slight 
j=l 
improvenmnt over the trivial bound. We also note that the central ideas of the proof 
stem from the work of Bentkus and G6tze [3]. 
In order to prove Lemnm 5.1, we first need to give another lemma, which is 
essentially a combination of two analogues of Weyl's inequality for exponential 
sunls over smooth mlmbers. We first quote these two analogues, essentially due to 
Vaughan and Wooley. as they are presented in [5] as Lenmms 3 and 4. respectively. 
Lemma 5.2. Let c and P be real numbers with P >_ 2. Define g(c) = g(c. P) as in 
(4.6). Fix a positive real number e. Then for su]ficiently small q. there is a positive 
real number3, that depends only on k such that either one bas Ig(a, P)[ <_ p-'r, or 
there are relatively prime itegers a ad q with q > 1 that satisfy 
g(a,P) « q¢P (q + pklqa--al) -W(2k) (logp)3. 

Lemma 5.3. Let c and P be real numbers wzth P > 3. Define g(c 0 = g(c. P) as 
in (4.6) with 0 < r/< 1/2. Fix positive real numbers A and e. Suppose that a and 
q are relatively prime integers with 1 <_ q <_ (logP) A and Iqc - ai <_ (logp)Ap -k. 
Th.en one has 
g(c,P) «, qP (q + Plqc - al) -W . 



2688 ERIC FREEMAN 

We now state the COlnbinatiol of these lmmnas. 
Lemlna 5.4. Define 7 = 7(k) as in Lemma 5.2. Fi.r positive real wumbers 0 and 
B'. Suppoe that P is a real umber with P  3. a,d that It is a al umber with 
Dee 9() = 9(o, P) as m (4.6), u,ith q s.uciedl small, ad suppose that oe 
(514) 19(,. P)I  ,P. 
Then lhere ei.sl. a positive itleger q and an ileger a wilh (a. q) = ] and 
q ççB',k,o I -k-kO and q - al ççB',k,O p-k-kop-. 

Proof. lt. is «lea.rly enough to aSsulno that we have 0 _< 1/2. \Ve apply Lellnlla 5.2 
with the «hoi«e « = 0/(2k). By (5.3) and (5.4), thele exist relatively prime integers 
ri ami q with q  1 su«h that t)ll(' has 
I,P 5 19(. P)I « qO/(2} p (q + pklq  _ al)-l/(2) (log p)3. 
It follows that 
q-e « p--2k(logp)6 and P]qo - a I « qe#-2k(logp)6k. 
By (5.3) and thc condition 0  1/2, we Cel'tainlv have 
('+a) 
q << (log e) qïç;) ,,« Iq - cri « (log )  -. 
Now we may apply Lemma 5.3. for large P. choosing A = 5k (B' + 3)/(1 - 0). 
say. and e = 0/(2k). XX obtain 
, « ¢/( ( + elv _ 1)-'/ 
It follows that one has 
q « p-kqe/2 and pk]q _ ai « p-kqe/2. 
Thus, since p  1 lnust hold. xve have 
2k ]l_k_k O . 
q «  2-o << and 
Thus the proof of Lcnmm 5.4 is complete. 
Now xvc are able to give the proof of Lemma 5.1. 
Proof. Suppose for the sake of contradiction that the condition (5.2) doe hot hold. 
Then there exist a positive real number e, an increasing seqnence of positive real 
mmbers P with lim. P = , and a sequence of real vectors ,  mr(P) 
with 
 nmv clearly ume that we bave e < 1. Bv trivial estimates, we have 
I(a, )l > 
Now we apply Lemma 5.4 to the sums gj(, P) =g (A)(), P) for ail suffi- 
cientlv large choices of n. For sufficientlv large t, wc have the bomds ç  P, and 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2689 
e > (log P,)-, and also P, > 3. Thus we nlay apply Lenmm. 5.4 with p = e and 
 = 1/k and B' = 1. Therefore, there are constants çl and c2 that depend only on 
k such that for large n an(t for 1  j  s, there are integers q,a and anj that satisfy 
(5.5) 1  qnj  C1 -k-1 and [Ai(a)qi -- anj[  C2e-k-lp2k. 
It follows ffoto these bomds and the definition (5.1) of mT(P) that we bave 
for ail j with 1  j  s, and all large n. 
For fixed e and T, we thus have that la.il and q,j are uniformly bounded. So 
there are onlv finitelv manv possible (2s)-tuples 
( qnl, qn2 ..... qns, anl, an2 ..... Ons ). 
Therefore one such (2.s)-tuple, say (ql ..... q.,a ..... a). oc('urs infinitelv often. 
Thus there is some sul)sequence, say {.f,}, wi/h 
(q  .... , qa,,s, ar,,, .... , (lfims) = (ql,---, q.ç, (il .... , 
forallmZ +. 
Since the sequence {ar,} is ('ontai,,ed within the compact set [0.1]x 
[-T, TI -, t,,«r i  fm-thcr subsequence {a,,} and a ve('t,)r a0 e [0.1]x 
[-T,T] n- such that 
Our goal in the remainder of the lemma is to show that for suciently large values 
of m, we bave a,m  (P)" which contradicts our original assumption. 
By (5.5) and the defining propcrty of the subsequence {h}. we have 
(5.6) IAj(a,)qj-aj<ce--lP - for l<j<s andforall mZ + 
Taking the limit of both sidcs of (5.6) as m goes to infinity, we obtain 
(5.7) Aj(a0)= a for 1 j s. 
qç 
Because condition (iii) of Theorem 1.2 holds, denoting a0 = (ara. a0,..., a0n) we 
nmst have 

(5.8) 
Therefore we bave 

O0(r÷l)  O0(r÷2) .... = OOR. 

A,-)(o0) __ __ai for 1 
qj 
Now, by (5.6) and (5.7), we have 
(5.9) [Aj (c, - 
= - -- + -- - A(o) < 
qj qj 
for 1  j  s and for all m  N+. Now, because A has full tank, we mav assume 
by relabeling variables if necessary that the submatrix A, defined as in (3.2). is 
nonsingular. Because of this and because the bound (5.9) hokls, in particular. 
for 1 < j < R, we must have  -0 < c3(F)e -k-lP-k for some constant 
c3 = c3(F) and for all m  N+. Therefore, by (5.8). we must have 
.10 , e [0.]   [_--l2,î,--,p--  o ,,, e z+. 



2690 ERIC FREEMAN 

If r = 0 holds, then for m suiïicientlv large, onenmst have ,m E M(Pn). But this 
contradicts our original assulnptiol that the sequence , satisfies , G my(P,). 
whence the equality (5.2) must hold. 
So we may assume for the remainder of the proof that 0 < r  R holds. Then, 
using (5.6) and (5.10), for m ff Z + and for 1  j  s, we have 
A(.r)/ )_ al Aj(n) ad  ija.ni (( e_k_lp_ k 
(5.11) 
3  'tre qJ qJ 
Since «41 iS nonsingular, there is an r x r SUblnatrix, say A0, of A that is nonsingu- 
la.t.  assume for ese of notation that Ao is the upper left-hand r x r subnlatrix 
of AI, noting that the other cses ail follow in the saine fashion as this case. For 
any real vector a = (al,... ,an), write a' = (a ..... a,.). By (5.11), we bave 
ol/ql [ U'ml 
T  a2/Ç2 
Or/qr tt'mr 
fbr some real vector w, = (U'l. tt',) with Iwl ŒEE e-k-lP -k Since we have 
assulned that A0 is nonsingular, we mav use Calner's rule to find b = (bi ..... br) 
with 
a/ql ] 
T a2/q2 
A 0 b = 
ar/qr 
Since A h integral entries, Olm moEv see that bi has the forln b = di/q for 
i < i < r, where dt is an integer, and q is a positive integer that satisfies 
(5.12) q  (qlq...q)det.(A)  c4(F)e -(+1, 
where the last bound follows from (5.5). 
 moEv aSSUlne, by reducing if necessary, that we have (dt, d2 ..... d. q) = I. 
By Cramer's rule again, we Inay find v  N with Av = wm, where we bave 
(s.a) lvml  «(V) -- 
Xit.e d = (dt, .... d.), and if d =  for sonm i. define di to be q instead. Then we 
have 
,' dq +v (modl) for mZ +. 
Now fix any choice of m large enough so that we bave 
(5.14) (og ,)   lnaX(«a(),«4{),()) -+1). 
Then by (5.10) we have 
 G [0,1]  x [-(log P)P -k., (log P)P-k] -=  
for this choice of m. Now write  = (d, d2 ..... d, O. 0 ..... 0), where there are R-r 
zeros here, and define ., similarly. Then, setting 
u =  + (0,0 ..... 0, a.{+x) ..... an), 
we have 
a=*d+u (modl), 
q 



S5 STEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2691 

where (dl, d2 ..... d,, q) = 1 and where, bv colnbilfil,g (,5.14) with (5.10), (5.12) and 
(5.13), we have 
l_<q_<(logP)B and u] (logP) . 
Thus recalling dcfinition (&ll), wehave ,  , (qd, P) for out particular 
choice of  and in fact recalling (.12), we also bave   .(P). As in the 
case r = 0, this is a contradiction, whencethe equality (5.2) must in fact hold. This 
completes the proof of Lemma 5.1. 

At this point, we make an observation about the lemma for those familiar with 
earlier argmnents of this type. VVe note that in previous work by the author ([10], 
[11]), the analogue of our Lemma 5.1 was proved with two different methods, for 
two subregions of the region roT(P). If we were to proceed by analog. with earlier 
arguments, we would instead have to treat a region mT, To (P), say, in place of mr(P), 
for positive rem numbers T0 with T0 _< T. The new region would be defined by 
mT,To(P)=mT(P) C {O : ]0] >_ T0}. 
Essentially by combining the arguments used for each region in previous proofs, we 
are able to dispense with the reqlfirement lai > T0. 
Having done most of the work, we tan now give a lemma that essentially savs 
that H IgJ(a" P)I is small for a E m. The idea of using such a lenmla is due 
j=l 
originally to Bentkus and G6tze [3]. 
Lemma 5.5. Deflne the forms Fi(x) as in (1.2) for 1 < i < R and the exponential 
suns gj(. P) for 1 <_ j <_ s as in (4.8), with r i sufjïciently small. Suppose that the 
coefjïciet matrix A associated with t]e sgstem F bas ratk R. Suppose also that 
the irrationalitg condition (iii) of Theorem 1.2 ]olds. Then tbere exists a function 
T(P) that depends only on B.! ad the coefjïcients of the forms F. F2 ..... F, 
that satisfles T( P) >_ 1 and 
(5.15) lira T(P) = 
P-cc 
and such that if we define re(P) as in (4.13) with this choice ofT(P), then one bas 

sup H Igj(,P)l : o(P) . 
om(P) 2î-: 

Pro@ The lemma is very similar to Lemma 6 of [10] and Lelnma 4 of [11], and the 
proof follows iii a silnilar fashion. [] 

We note that this lelmna (and Lemlna 5.1) holds for a.ny positive choice of B, but 
that the function above that is o (P) depends on B. We have stated this lennna 
in a general fashion in the hopes that it nmy be useful for future workers. 
We observe that one could ensure that the function that is o (P») depends only 
on B, r] and 2R - r of the coeflïcients. This follows, with some effort, after finding 
a subset O r C_ {1,2 ..... s} with IJI --= 2R - r such that the conditions Aj(o) ( Q 
for j  J, taken together, imply that c+l ..... cu = 0. This can be proved, 
although out method of proof, at least, is hot straightforward. 



2692 ERIC FREEMAN 

Iu the remainder of the paper, we fixa flmction T(P) that satisfies the conclu- 
sions of the above lelnma. We note that this is the special function we referred to 
above in section 4, and is used to define the lninor arcs and trivial arcs. 
We observe at this point that we could obtain corresponding results which are 
very similar to Lemmas 5.1 and 5.5 if the expouential sums gj were replaced by 
exponential sums over a complete int.erval. The only major change needed would 
be to use Lemma 2 of [11] in place of out Lelmua 5.4. 

O. THE lk|INOR ARCS 

In this section, our goal is to show that the coutribution from the minor arcs 
to the integral in (4.9) is o (P-nk). We first give a lemma, which is essentiallv a 
restatement of results due to Vaughan [19]. [20], and results due to Wooley [23]. 

Lemma 6.1. S-uppose that k is an nteger with k >_ 2. Define g(c) as in (4.6), 
with q sufficiently small. Then there is an absol-ute positive constant ff' such that 

ff t is a real number satisfying either 
(i) t >_ miu (2 , L'(log/," + h»g log k + 2) + 
\ 

OF 

then one ha 

C'k log log k ' for k > 3. 
log k J - 

(il) t > 4 for k= 2. 

(6.1) ff01 

ig(c01 t , pt-e. 

We observe that one could certainlv improve Oll the lemma iii certain cases, but 
we choose to use only the above bounds for out results. 

Pro@ If the first bouud of coudition (i) holds, then the result is Lenmm 6 of [11], 
which is essentiallv due to Vaughan [19]. [20]. If on the other hand. the second 
bound of condition (i) holds, then we may essentially quote Lenmm 7 of [11], which 
itself follows ahnost immediatelv from work of Wooley [23]. We note that the 3 in 
Lemma 7 of [11] has been replaced bv a 2 here; I aih gratefifl to Scott Parsell for 
showing me the technique one uses to make this inlprovement. 
In the case in «hich (ii) holds, we give a proof for completeness. Define 

e=t-4. 

VCe need only prove that Olle has 
 [g(a)[4+«d a « 
p2+«. 
Clearly. we may assmne that e _< 1 holds. For convenience, we write 
2 

Defiue 

91 = rl ° G [0.1]'lg(cOI > P(log p)-C;}. 
Also, for positive integers m. define 



S_ STEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2693 

Now for o E -qlm, we apply Lenmm 5.4 with the choice/? = G. Thus, for large 
P and any positive real number , there exist coprime integers a and q with q _> 1. 
and 
<< 2 m(2+6) and ( - î << q-12m(2+6)P-2. 
q 
q 

Thus ,are have 
It follows fol"  < e/2 that 
On the other hand. one has 

(.3) 

But ri0.11 

q 
q2rn2+5) a---1 

« p2+e  2-m(-26) « p2+e. 

I9(a,)lada is less than or equal to the nmnber of sohltions of the equation 

with 1 _< :ri _< P fol" 1 < i < 4. This is bounded bv a constant nlultiple of p2 log P, 
a well-known result, which can be proved bv elementarv means. Thus ff'oto (6.3) 
and the definition of 9l. we have 

[0 Lq(°)[4+¢d  (P(log P)-) p2 log P « P:+¢ (log p)-l, 

by our choice of G. Combining this bound with (6.2) completes the proof of Lenmm 
6.1. [] 

We note that (6.1) is an example of what one might call an "'exact Hua inequal- 
ity". In most work using tlle Hardy-Littlewood method, one uses bounds of the 
type (6.1) where one onh" needs to show that fol" anv « > 0, the left side of (6.1) 
can be bounded bv pt-k+¢. Dispensing with this e is crucial for our work. The use 
of such an inequality stems from the work of Bentkus and GStze [3]. 
For the renlainder of the paper, we now fix a choice of  so that Lemmas 5.5 and 
6.1 hold fol" this choice. Now we turn to what is essentiallv our analogue of Hua's 
inequality. It is very similar to Lemma 8 of [10]. 
Lenmm 6.2. There is an absol,de positive real constater C_ with the followin9 
property: 
Assume that the forms F1. F2 ..... FI are as in Theorern 1.2. with coejïjïciet 
matrix A satisfyig (3.2) and (3.3). Assume lhat f is a positive integer satisfging 
(>_5 for k=2. and 
( 72k l°g l°g k )fork>3. 
g >_ nfin 2  + 1. k(log k + log log k + 2) + log t,- - 



2694 ERIC FREEMAN 

Define the exponential smns gj(o, t 9) as in (4.8), and deflne the function K as in 
(4.2). Let d(P) be a nonnegative real«,alued.funetion, and let n be any subset of the 
region 

Also, define 

Proof. Observe first that for an3" real number e with 0 < e < 1, one has 
s R 
L II 'gJ(«'P)' 1-I 
j=l z=r+l 
s 
« sup lgj(. P)l 

IK(ai)[da. 

It follows fFOlll trivial estiinates that Olle has 

 R 
j=l i:r+l 

« (h(n" P)P») (P»-gn)- L 

eR R 
H IgJ(«" P)II-e H 
3:1 i=r+l 

We may certailfiy choose a positive real number e so that we have 
(6.4) g(1-e)>4 if k=2. 
Defining C' as iii Lenuna 6.1 and choosing OE2 to be sufficiently large, we lllay ensure 
t hat we have 
( C'kloglogk ) 
.t _> lnin 2 k + 1. k (log k + log log k + 2) + log k + 1 if k _> 3. 
Thus we ma.v choose a positive real nulnber e, slnall enough (in terlns only of k and 
C) so that we have 
( C'k log log k) 
(6.5) g(1 - e) > min 2 . k (log k + log log k + 2) + log k if k > 3. 
In each of the cases, we denote our particular choice of e by u. Now one can join 
the proof of Lemnm 8 of [10] after equation (66), and then follow the remainder of 
that proof with only slight adjustments. The bounds (6.4) and (6.5) are the crucial 
bounds that, we need to apply Lemnm 6.1. V'e omit the details. [] 

Now we can wrap up our work on the minor arcs. We have the following lenmm. 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2695 

Lemma 6.3. Suppose that we are in the sctting of Theorem 1.2 and that the 
coejïficiet matrix A of the system F satisfles the corditions (3.2) and (3.3). Choose 
a function T(P) as ir Lemma (5.53. Define the exponertial sunrs gj(o) as in (4.83, 
with  sufficiently srnall, the region m as in (4.133, ard the function Ix" as in (4.23. 
Then one bas 
s R 
Jm = "3(0)I i=r-t-lH I'(°zi)[d°z =o (tgs-'k). 

Pro@ We siinply apply Lenuna 6.2 with the choices n = m and d(P) = 0. VVe ha.ve 
h(m, P) = o(1) by Lelnma 5.5. Tlms the proof of Lenllna 6.3 is coniplete. [] 

7. THE TRIVIAL ARcs 

Iii this section we show that the contribution frolll the trivial ai'es to the integral 
in (4.9) is o (ps-R), which is now easy to do, having done the necessary work 
above. We have the following lenlnia. 

Lemma 7.1. Suppose that we are ir the setting of Theorem 1.2 ad that the 
coejïficiet matriz A of the system F satisfies (3.2) and (3.33. Choose a functior 
T(P) as in Lemrna (5.5). Defl, e the exponential sums gj() as in (4.8) with q 
suciently small, the region t as in (4.143. ad the functio Iç as in (4.2). Then 
one has 
s R 

Pro@ YVe apply Lenuna 6.2 with tlie choices n = t and d(P) = T(P). 
h(m, P) = O(1) by trivial estiinates. Thus we obtain 
R 
fil IgJ(°t)l H IIx-(°q)'d° «7. (T(P))-IP s-Rk, 
= i=r÷l 
which by (5.15) of Lemma 5.5 is o(PS-Rk). 
complete. 

We have 

Thus the proof of Lemma 7.1 is 

8. THE IklAJOR Aacs 

We now treat the lnajor arcs. Our goal is to show that for large P we have 

s R 
3----1 i=r÷l 

8.1. Approximation on the major arcs. We start our treatment of the major 
arcs by approximating the functions gj(o) by auxiliary functions. We need soine 
notation before we do so. We define Dickman's function p by the conditions 

(8.1) 

p(,) =0 
p(u) ---- 1 
up'(u)-----p(u- 13 
p is continuous 
pis difïerentiable 

for u_<0, 
for0<u< 1, 
for u > 1, 
for u >0, 
for u > 1. 



2696 EIRIC FREEMAN 

Also, for real nulnbers ff, define the functiol 
fo Pk Ç log_x_  
1 oc(l/l)_lp e(:r)dr. 
(8.2) () =  » og P ] 
For real vectors   Nn, define Aj()  in (4.7), and write 
(8.3) wj() = w (Aj()) for 1  j  s. 
Also, for integers q and a with q k 1, define 
= k q 
We now collect sonm results, given by Brfidern and Cook [5], in the following 
1PIllllla. 
Lelnma 8.1. Defit, e 9() as i (4.6) and ([) as i (8.2). Suppose ha¢ a and q 
are iegers with q  1. ad ha  is a real number. The one bas 

and 

oe(fl) « lnin (P, 

Proof. The first result is simply equation (29) of [5]. The second result is essentially 
the third centered equation on page 135 of [5]. [] 
We note that, as remarked by Brfidern and Cook, a and q are hot required to 
be relatively prime. We now state the central lemma of the section, which is ver3" 
similar to Lemma 4.4 of [16]. 
Lemma 8.2. Suppose that we are in the setting of Theorern 1.2 and that the 
coefficient rnatriz A of the system F satisfies (3.2) and (3.3), and that there is a 
real vector z satisfying (3.6). Define the so-called singular series  by 
(3 = 1 r = 0, and 

(8.5) @:Z Z q-S H S (q, AJr)(a) ) for 
q=l a (nod q) j=l 
(a ..... a,.,q)=l 

and the singular integral Z(P) by 

Fiz any positive real number e. Then if P is a sufficiently large positive real number. 
oe bave 

« p-n," ((log p),(+,)- + (logP),(-(e/k)+,)). 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2697 

PTvqf. There are three steps of the proof. One first at)proximates each fimction 
bv terres of the form q-lS (q, A.T)(a))coj(/)on each of the major arcs, then 
gj( o) 
one extends the integration over each nm.jor arc to ail of IR R, and then one extends 
the smn over q to all positive integers q. The argument closely follows the proof 
of Lemma 4.4 of [16]. One major difference involves the use of the approximations 
given in Lemma 8.1. Since we are using exponential smns over smooth numiwrs, we 
need to use these approximations instead of mre standard results for exponential 
sums over complete intervals. Finally, we observe that the only condition we really 
need on f for the purposes of this lenuna is the b,mnd f >_ 2/,: + 1, [] 

Now we turn to consideration of the singular series  and the singulm" integral 
Z(P). In particular, we shall show that we have 9 » 1 and also that xve have 
Z(P) » P-Rk for sufficiently large P. We first treat the singulm" series. 

8.2. The singular series. \'e give some (lefiuitions. Sui»pose that G1, G2 ..... Gr 
are r integral diagonal fOllllS iii s vm'iables with coefficient matrix B, with entries 
dij. hl Section 8.2, we consider integral forms, and of coin'se we aSSllllle that r _> 1 
holds throughout this section. 
\Ve then define the singular series (G} associat.ed with this svstem of r finms 

by 

(8.7) 

where we set 

e=e(.¢)= E q-* H S(q, ab 
q=l a mod q) 
(al ..... a,.,q)=l 

r 
aba)=Edijai for 1 <_j <_s. 
"We return to the notation of the rest of the pat»er for a momcnt. Observe 
that the definition of  given here coincides with the dcfinition (8.5) in the case 
r = R. Moreover, in general, whcn 1 ç r 5 R hokls, the first singular series is 
exactly the latter singular series, where the latter is associated with the first r forms 
F1 ..... F. Note also that the first singular series is independent of the R- r fonns 
Fr+l ..... F. 
Suppose that p is a prime and , is a positive integer. Then we sav that an 
integral vector x = (1, x2 .... , ,) is a solution of rank r (mod p) of the svstem 
of congruences 
(8.8) Gl(x) 
if there is a subset J ç {1, 2 ..... s} with ]JI = r such that one has p  det(Bj) and 
p {  x. Also, for anv prime p and any positive integer n, we define M (p, G) to 
jJ 
be the number of solutions x (mod p) of the system (8.8). 
Now we shall define the concept of a nonnalized systeln of forms.  follow 
Low, Pitman and blff [13] closely, but we need a slightly more general notion. 
 essentially want to define a notion of a system such that a related system, 
which results after setting all but some subset of tf of the variables equal to zero. 
is normalized in the original sense of Low, Pitman and kXlff. 



2698 ERIC FREEMAN 

Suppose that the coetï-icient matrix B contains t disjoint nonsingular r x r subma- 
trices BI, B2,..., Bt. To be clear, t)v this we lnean that there is solne permutation 
of the columns of B so that the first r columns form a nonsingular submatrix, the 
second r cohnnlls form a nonsingular sut)nmtrix, and so on, through the t th set of 
r cohmms. We define 
Now let j = (j, j2 ..... jt) t)e the ordere(l (tr)-tuple su«h that the particular matrix 
[B1B2... Bt] is the sut)matrix of B consisting of the columns of B indexed in order 
t)v j,j .... ,j; that is, we define j so that for 1  v  t and 1  b  r. the 
-th cohmm 
]t,(r-l)+h 
dcpcnds on j. Also set 
. = {j. j,..., j,}. 
Suppose that p is a prime dividing A. Here we define, following [13] closely, 
a p-operation on the fornls G1 ..... G r aS a transformation that produces integral 
forns H .... , H, and has the fidlowing steps: 
(i) Pre-multiply B by an integral unimodular matrix U with entries in the set 
{0. ..... 
(il) Next, multiply at most tf" - t" of the cohuims of U.I by pk and multiply 
anv of the colunms of U{1 ..... s}kJ by pk; 
(iii) Then divide g of the rows by p. where we have 1 N g  r. 
As discussed in [13], step (i) corresponds to adding linear combinations of some 
of the forms to one or more of the other forms. Step (ii), on the other hand. 
corresponds to writing xj = pgj in each cohuun j that one multiplies by pk, and 
then trying to solve the new inequalities in the variables yj. Step (iii) corresponds 
to dividing g of the r equations by p. One can check.  in [13]. that a p-operation 
is possible for all primes p that divide 
Note that for the resulting system H. we bave 
for sonle ilteger m.  say that such a p-operation is permissible if one h 
m<0. 
Observe that. upon perfornling permissible p-operations for anv of the primes 
p dividing A(G), we can find a system H that can be obtained from the original 
system G via a finite sequence of permissible p-operations and such that A(H is 
minimal. If G is a system of r integral forms as above, such that A(G) cannot be 
reduced by any permissible operations, then we say that G is a (j. t)-norlnalized 
system. Finally, if G is a system of r integral forms in exactly tr variables, then 
we simply say that G is a normalized system.  note that. in this case, our 
definition clearly agrees with the definition given for a normalized system in [13]. 
e make one other observation. Suppose that.  above, the coecient matrix 
B of a system G, in s variables with coeNcients dij, contains t disjoint nonsingular 
r x r sublnatrices B, B,..., Bt, and define j and J as above. Then. we define the 
system G* in tf variables, by defining, for y  Z t and for 1 < i < r. the forms 
(8.9) 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2699 

Note that this is simply the system obtained bv setting ail variables with indices 
j  J equal to 0, and subsequently reordering the variables. Observe that the 
coeffiçient matrix B*, sa)-, of the system G* has the form 
B*=[ B, B2 Bt ]. 
Suppose nov« that a system H can be obtained rioto the system G airer a finite 
sequence of permissible p-operations, and suppose that H is (j. t)-normalized. Then 
consider the system H*, defined as in (8.9). We can sec that the saine p-operations 
(restricted to the columns j E J) allov« one to obtain H' from the system G*; airer 
all, the variables of G* are a subset of those that appear in G. Each operation is 
certainly still permissible, since the definition of A involves only the coluinns j E J. 
If one could reduce A eH') via a permissible p-operation, then we could simply 
extend step (ii) and nmltiply all of the cohmms of UB{ ..... s}\J by p#. This would 
give a permissible p-operation for the system H, which contradicts our assmnption 
that H is (j. t)-normalized. So if H is (j. t)-normalized, tllen it follows that H' is 
normalized, and moreover, if H results from G airer a finite sequcn('e of permissil»le 
p-operations, then H' results from G" ri'oto the saine sequence of p-operations. 
We can now sIate the following lemma, which is a step towards bomding the 
singular series below. We do slightly more than we need to, in the hope that it vill 
be usefifl for fiture workers. For this reason, we state I he lemma in a self-contained 
lllallller. 

Lemma 8.3. Svppo.e that r, k and s are positive integers with k >_ 2. and suppo.e 
for l < i < r that 
D,(x) : dilfl" Î ÷ d,22"  +... + dis.r  
fs an integral diagonal form of degree k. Suppose that s satsfles s k tr. where t fs 
an iteger satisfying 
t>2k+l. 
Suppose that the coecient matrix C of the system D contais t disjoint nosin9ular 
r  r submatrices C1, C2 ..... Ct. and deflne 
t 
A(D) =  
Also, define the sig.ular series (D) as m (8.7), Suppose that the followmg prop- 
erty, which we denote by Pet, k, r), holds: 
Given ang system of r itegral diagoml forons G1. G2 ..... Gr of degree k in 
tr variables, with coeciet atrix B which consists of t disjoint osingular 
r  r submatrices and such that the system G is ormalized. then for every 
prime p and every positive integer n. there is a solution x of tank r (mod p) 
of the system of congrueces 
Gl(x) G2(x) --- oeGr(x) 0 (nlodpn). 
Then the series (D) converges absolutely and one bas 
(D) >>D 1. 
If one also bas 
t>kr+k+l, 



Proof. We first give some more notation. For anv prime p, define 3' = "(k.p) bv 
«hoosing - to satisfy p'-II/,', and settiug 

{ 1 if T=0 
-) = "r+l if 7->0andp>2 
T+2 if "r>0andp=2. 

Also. for anv l)rime p. we definc 
(8-J 1)) \D(P) ---- 

.'il (p'. D) 
__,oe pn(s-r) 
As iii Q h«l|)t( r 5 of [61, and ll,'qing al,,qo Lenllna 2.1() of [15] OllC lllV see that this limit 
exists, that (D) converges al)solutely, and that @(D) is equal to an absolutely 
('(llV('l'gellt l)r()(luct, that is. we have 
(.) (D) = H 
P 
(XXb noie that it is in his argument that one uscs the condition t k 2k + 1. and 
that he rate at which the product converges deI)Cnds on D.) 
Now dcfine D as in (8.9), and dcfine j and J as in the discussion above. 
may find a (j. t)-normalized svstem G. which can be obtained from D alier a finite 
sequence of I)ermissible p-operations. As we bave noted above, G* is then a nor- 
malized system, which ont obtains from D* aftcr (essentially) the saine permissible 
p-operations. 
Since proper/y P(t. k. r) holds, there is, for ail p and -n, a solution w of rank r 
(mod p) of the svstem of congruences G*(w)  0 (mod p). Bv the way G* w 
dcfiued, one can s that if y = (y, ye ..... y) is defined bv 
{ wj if j  J 
.v = o if j C j. 
/hen we have that y is a solution of tank r (mod p) of the svstem of congruences 
G(y) oe 0 (mod p). In t)articular, this holds for n = 7- XX mav thus apt)ly Lelnma 
6 of [131 to the system G. whence we have 
(8.12) 5I (p,G) k p(»-)(-) tbr n > 7- 
Froln this fact we will deduce a lower bound for 5I (p, D). 
To this end. suppose for some positive integer r tha.t y  Z  is a solution of the 
COllgrHellCeS 
Gi(y) Ç2(y)  ""Ç,-(Y)  0 (mod pn). 
Recall that the svstem G resulted from D after a finite sequence of permissible q- 
operations. Let H be a svstem such that G arises from H after a single permissible 
q-operation. Let I be the subset of {1, 2 ..... s} consisting of the cohmms affected 
bv step (ii) of this q-ot)eration, that is, let I consist of the indices such that the 



SYSTEIVIS OF DIAGONAL DIOPHANTINE INEQUALITIES 
corresponding cohmms in step (ii) are nmltilflied by q. 

x = (Xl,X2,...,a's) by setting 

We show that one has 

{ qyj if j C I 
a if jI. 

2701 

Then define the vector 

(8.13) H, (x) = Ha(x) =.-- --- Hr(x) _= 0 (mod 
To see this, let H (0 and H (i) i,e the syst.ems that result a.fter steI)S (i) and (ii) 
of the q-operation, respectively. We certainly have H¢ii)(y) = 0 (mod t/): indeed. 
sonle of these forlns are congruelt to 0 (lnod pq). Then observe that we bave 
H(ii)(y) = H(i)(x), whence H(i)(x) 0 (lnod p) holds. Since the lnatrix U is 
unilnodular, so that in particular its deterlninant is hot divisible by p. one has that 
(8.13) holds. Thus any solution y of G(y)  0 (mod p) gives lise to a solution x 
of H(x)  0 (mod p). If q  p holds, we therefore have 
M (p",H) k Al (I,',G). 
If q = p holds, we might have some reduction in the nulnber of solutions, because 
lnultiplicatiol by p in /pn has kerlml of size p. but we certainlv bave 
M (p", H ) 
So, by repeating this analvsis for each permissible q-operation, one can see that 
if G is a (j. t)-normalized system arising from the svstem F after a finite sequen«e 
of permissible q-operations, then one has 
M(p,D) OE if p'[[(D). 
pro(s-r) 
Now the linfit k(P) exists; this follows in nmch the saine wav as the corresponding 
fac for D. It follows that we bave 
(8,1) k(P)  p_) if pH(D). 
It follows from (8.12) and the definition of k(P) tiret for ail primes p we bave 
(8.15) kD(P)  p-(v+ord(aD)))(-). 
Since the product  kD(P) is absolutelv convergent, there is a constant c(D), 
which mav depend on D, such that we have 
1 
Thus, using also (8.15). we have 
1 1 
pNc(D) pNc(D) 
1 
 (A(D)) -  p-(- »D1. 
pSt(D) 



2702 ERIC FREEMAN 

Now suppose that we have t >_ kr + k + 1. One can prove as iii chapter 5 of [6] 
that for primes p, one has 

(8.16) 
n=l 

where for positive integers , we define 

s (p') = 

Z H S(p''Mj(a))" 
a (mod p') j=l 
(al ,...,a,p)=l 

Now suppose that p{ A(D). Sut)pose for 1 _ v  t that Cv consists of the 
cohmms j,,jv .... ,j, in that order. Then for a satisfying (al, .... a,.,p) = 1, 
there nmst exist sonle j  {j, ..... j. such that p  ]iii(a). since we have p { 
det(C). Thus for any prime p with p  A(D), and any positive integer , by the 
standard estimatc S (p', a) « p,(1-(1/¢)), which holds for (,p) = 1, we have 

S (p') << p- 

Z pnS-(nt/¢) « pn(r-(tl¢)). 
a (mod 
(al ..... 

Combining this last bound with (8.16) and using t >_ kr + k + 1 yields 

\D(P)- i ŒEE Zp nr-(t/k)) ŒEEp -1-(1/k) for p{A(D). 
n=l 

So there is a constant C that depends only on k, r and t such that one has 

(8.17) I\D(P) - 11 < CP --(/) for p{A(D). 

Now, because Z Cp-I-(1/k) converges, there is a constant C that depends onlv 
p 
on k, r and t such that one has 1 - C -1-(1/t¢) > 0 for p > C and 

1 
H (1- Cp -1-(1/k)) _ . 
p> 

Now for all p we have \D(P) > 0 ff'oto (8.15); so by (8.17) we have 

plA(D) p'A(D) 

plA(D) pA(D) 
1 
 I-[ (1 I-[ D()" 
p[A(D) 



SïSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2703 

It follows from (8.15) that we have 
1 
p p<(, 
p/XID) 

H 

p--('+ord,,( A(D) ) )(s--r) 

(A(D))T-* H P-«-) 
plA(D} 

(A(D)) '-* (A(D))20 "-) " 

This completes the proof of Lemma 8.3. 

Now we give another lemma that builds on the above lemma and completes the 
treatment of the singular series for the cases k _> 3. As in the case of Lemnm 8.3. we 
do slightly more than what we will need, and we stale the lemlna in a self-contained 
fashion. 

Lemma 8.4. Suppose that r, k, and s are positive integers, ad suppose for 1 <_ 
i < r that 
Fi(x) = Aixî + Ai2x2 +... + Aixs 
is an integral diagonal form of degree k. Suppose that the coefficient matrix 4 of 
the syste F contains g diooint nonsingular r x r submatrices .4 ) A () AW) 
where g is a positive iv teger satisfgig 
fk2k+l. 
«  = H ]dot (A}:')]. Define  = (F)as in (8.7). Finally, suppose that 
one of the two followin 9 statements holds. 
(i) k is odd, and k  3 holds, and one bas g  kmo, where mo is the least 
positive integer m such that one bas 
(ii) k 2 3 holds, and one bas 
g > k [48k = log 3rk =] . 
Then oue has 
O(F) » 1. 

Moreover. if 

g>kr+k+l 
holds, then there exists a positive real constant c(k, r, s) that depends only 
o k, r and s such that one bas 
rg(F) _> c(k, r, s)A -. 
Moreover, we note that if k is odd with k >_ 3. then there exists a absolute positive 
real constant C such that condition (i) holds if one has 
rk log 2/," 
(' > + Ck log(r log 2k). 
- log 2 



2704 ERIC FREEMAN 

Proo.f. The last statement of the lemma can be checked with a straightforward 
computation. 
Thus to prove the lemnm, we need only check that the condition P(& k.r) of 
Lemma 8.3 holds for out choices of (. So suppose that G is a normalized system in 
l'r variables. For the case in whi«h condition (i) holds, we may apply Theorem l(ii) 
of [13] to see that for any positive integel" 71, the svstem G(x) = 0 (mod p') has a 
solution x of tank r (mod p). On the other hand. for the case in which condition 
(ii) holds, we may apply Theoreln 3(i) of [13]. 
Tlms in either case, the condition P(& k, r) of Lemma 8.3 holds, v«hence Lelnma 
8.-1 follows. 

Now wc give a lclmna to treat the singular series in the case k = 2. \Ve observe 
that one could surely obtain a rcsult that is better for large r, but we choose hot 
to pursue this hcre. 
Lemlna 8.5. Suppose lhat r and  are positi,e bdeger.s, and suppose for 1 < i < r 
that 
Fi(x) = )il.Fî -- )i2J'72 --...-- 
i.s an itegral diagonal quadratic form. Suppose that the coeJficient matrix A. defined 
as in (1.3). contams g disjoint 71oT,s'ilgular r × r submatrices, where  is an integer 
sati.sfying 
t' OE lllill (-11.2 + 41" + l. 38-1 log 161" + 5). 
Define t = t(F) as i'n (8.7). Then one bas 

5(F) »F 1. 

Pro@ If the first bound for  holds, then it follows that any nontrivial complex 
liniar combilation of the forlns F ..... F bas rank at least 4r 2 + 4r + 1. Bv the 
theorem in [18], the singular series t is positive and depends onlv on the forms 
F1, F2,.... F,-. One can readilv check tllat the singular series t is defined in [18] 
in the saine ruminer as we have defined it. 
Sul)l)ose instead that the second bound for  holds. \Ve give a sketch of the 
proof in this case. We first seek an analogue of Lemma 12 of [13] for the case 
k = 2, with m_>[1921og16r+2]. For primespwithp<_81.2 , say. it is easv to 
check that Lenmm. 5 of [7] provides an analogue of the desired type. To obtain 
an appropriate analogue of Lelnma 12 of [13] for primes p > 8r 2, one applies an 
adaptation of Theorem 2 of [14], «ith, say, c = 4: one can check that if one assumes 
that the matrix of coefficients contains c + 1 nonsingular i" × r submatrices, rather 
than assuming that the matrix is highly nonsingular, then the result still holds. 
(This can be seen by noting that the inequality q(B) > ci, which would still hold, 
is the key condition needed on page 339 of [14].) 
In either case, we bave an analogue of Lemma 12 of [13]. and thus ce can show 
that condition P(tç, k, r) holds, a.s in the proof of Theorem l(ii) of [13]. [] 

Michael Knapp and Professor Wooley provided me with a proof of a result closel3 
related to the second part of thc above lemlna, for which I alll grateflfl. 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2705 

8.3. The singular integral. F{ecall that in (8.6) we defined the singular integral 
Z(P) 
-(P) = . Çj=IIlt'UJ(fl) ) (i=rl[Ç(Çi)) d" 
Our goal in this section is to demonstrate thN for large positive P we have the 
bound 
(8.18) g(P) »F P-- 
Instead of using the traditional approach which uses Fourier's Integral Thcorem, 
we use a method given by Schmidt [18]. Below we [ollow parts of [18] very closely. 
Much as in [18], for anv positive rem mmbcr T and anv rem numbers a and , 
we define 
2 
lçT(O)=lç(aY 1)= ( sill ( -1) ) 

l.lld 
(8.19) 

(8.o) 
We now define 

= j" T(1-Tlfll) for Ifil-<T-1 
¢'T ( fl ) 
0 for [fl[ > T -1 
From (4.3), one may readily deduce the %llowing identity, which holds for all rem 
nmnlmrs , namely, 
@T(fl) = [" e(al3)Kr(a)d«. 

By (4.4) and a similar bound for tçT(a), the integral converges absolutely for each 
choice of P and T. 
We shall see that for fixed P, we bave liln ZT(P) = Z(P), and we will also show 
that for large T. we have ZT(P) »F p-Rk. These two facts together establish the 
bound (8.18). To prove the first fact, we give a bound for the difference ZT(P) -- 
Z(P). 
Lemma 8.6. Sppose that T ad P are positive real numbers with 
P >_ 1. Sppose that R. r. I; and s are bte9ers with R >_ 1.0 < r < R ad k >_ 2. 
S'uppose for 1 < i < R that 
Fi(X) = ,il,Z'Î ÷ ,i22F22 ÷... ÷ 
is a real diagonal form of degree k and that for 1 < i < r, the form Fi is itegral. 
Asstrne that one bas IIF]] >_ 1. &tppose also that the coeJficient matrix A of the 
systern F is as in, (3.2) ad satisfies (3.3). where one bas 
g,>k+l. 
Define Z(P) and ZT(P) as in (8.6) a«d (8.21), respectivel. Then ote bas 



2706 ERIC FR EEMAN 

We note that the implicit coTstont in Vinogradov's notation bere depends af most 
on H. r. k and s and, in particular, does hot depend on the coeciets of F. 
Pro@ Observe first that, in the case r = 0, we havc Z(P) = ZT(P). So we can 
assume that we have r  1. It follows from the definitions (8.6) and (8.21) that we 
have 
From the pemfltimate centered equa.tion on page 305 of [18], one has 
1--HIçT()«T -e max [e«T-el] e for 1[ <T. 
i=1 
F 
and for ]] OE T, one clearly has  lç() (( 1. çoml,ining these 1)ounds with the 
estimatc for w() given in Lemma 8.1, and thc 1)(rond (4.4) [or Iç(o). one h 

Consider the first integral on the right-hand side of (8.22). By H61der's inequality, 
one has 

(8.23) 

j=l 

« H ,<T ]/]2 H lllill(P']AJ(t)l-l'k) gdt 
v=l j=(v--1)lR + l 

Fol" a fixed choice of v with 1 < v _< & one Inakes the change of variable "7 = 
(71 .... ,TR) = U,,(/3) given by 7j = A(,,-UR+j(/3) for 1 _< j _< R. and obtains 

j=(v--1)R÷l 

R 
'u-l(') 2 H 111i11 (P, Ij[-1/k) ' 
3=1 

If "7 = U. (/3), then one has 
l'ri = IUv()l < IIFll- 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2707 

and by Cramer's rule and Hadamard's inequality, one has 

(8.24) 
It follow8 that we lmve 

(8.25) 

AI' 

But one has 
SI-ri< RIIF[[T 

d7 

whence we have 

R 
j=l 

since we have 7 > k + 1 and IIFIIT _> 1. 
Thus, by (8.22), (8.23) and (8.25), we have 
2c(P)- 2eT(P) << IIFII2ST-11p s-sk 1-I A73/g 
+ Ps-Rgf, O H rein (P, IAj(fl)] -llk) dfl. 
I>_Tj=t 



2708 EHIC FREEMAN 

By Hiilder's inequality and a change of variable as al»ove, we have 
:Z(P)- :Zr(P) << IIFll2"T-1/"P s-' H AÇa/e 
,(/,.  
r=l I'l--cFa'T j=l 

where CF,v is a positive constant which by (8.24) we lnav defilm by 
«œe,v = RR/=IIFI]R_ l- 

d7 

since we have t e >_ k + 1. Combining the last txvo bounds completes the proof of 
Lclllllla 8.6. [] 
Now we prove a lemlna which states that for T a.nd P sufficientlv large, the 
quantity ZT(P) is bounded below. 
Lemma 8.7. Suppose that we are ir tbe selti9 of Tbeorem 1.2 and that tbe 
coefficient rnatriz A of the sgstern F satisfies (3.2) and (3.3). and that there is a 
real vector z satisfying (3.6). Suppose also that one has IIF[[ > 1. Define At as in 
(3.3). Suppose that T aM P are real vurnbers satisfying T > 1 and 
(12 )1/ 
(8.26) P > -'n/?2R]lF[[2/lnax(A-2,1 ) 
where 6 is as in (3.6). Define 2-r(P) as in (8.21). Then there is a constant ci = 
q(F,k, R.r,s, 5. z) that does hot &pend on T or P such that one bas 
çT(P) OE Cl Ps-Rk. 

Pro@ Recalling the definition (8.3) of wj(13), we can write the absolutelv conver- 

,=i)1 ( og«  
r 
t=l i=r+I 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2709 

Using the identities (8.20) and (4.3), raid recalling the definition (3.4) of Li(x), 
we can rewrite ZT(P) as 

1 )l/k-1 IïI p \klogP 
- ,pkls(.rl-.-Xs )I*',T(Li(x)) 1Ç[ g'(Li(x))dx. 
j=l t=l i=r+l 

ZT(P) is certainly larger than the corresponding integral over the slnaller region 
__.pk , noting froln Lelnlna 12.1(i) of [21], alM (8.1), that the functiol p 
is always nonnegative. Fol P satisfying (8.26), one certain|v has P > (5/2) -l/k, 
6pk 
-- < xj < pk, it follows ff'oto (8.1) that one has 
whence for xj satisfying 2 - - 

Ç log Xj " 
 ] = - 

So we have 
z.(P) > ,;. .] 
Now define T4p.T to be the region 

min\l'A//'() fol'l</<," and 
37" 

 ( ) } 
IL(x)l < mil, 1.A'l/ fort + 1 <z < R 
It follows froln the definitions (8.19) and (4.3) of vr(a) and b(a), respcctiveh, that 
we have 

(8.2/) 

T(P) 2> ps-skTr]t s (TPp.T), 

where Ps denotes s-dilnensiolml Euclidean lneasure. 
We now lnake the linear clmnge of variable w = V(x) given bv 
Lj(x) for 1 _< j <_ R 
w j= .r for R+l <_j <_s. 
Since A1 is nonzero, we can see that we have 

(8.28) /£s (7-P,T) >F //s 
where we define $.7 to be the region V(74p.T). Note that $ T is the set of w 

[î ] 
such that there exists an x 6 , Pk 

with w = l(x) and such that one has 

lnin(1./N}/R) lnin(1,zll/R) 
I**'1 < 3T fol. 1 < i < r and Iwl < 3 

for r+l < < R. 

Now we give a lenuna, which is essentially due to Nadesalingaln and Pitlnan. 
(SeP [16], Lellllll& 5.2.) 



2710 ERIC FREEMAN 

Lemma 8.8. Let R and s be positive integers satisfying s > R. Let 
A = 
be a real R x s matrix. For 1 
Let A dcnotc the absolu.te value of the detetinant of the lefl-hand R x R subatrix 
of A. Suppose that we bave A > O. Additionallg. suppose that Q is a real number 
satisqing 
(8.29) Q 
Suppoe also that w,..., wn are çeal number satisfying 
]wl< for l < i < R. 
- 3 
Let 5o = So(W,..., wn) be th« set of all real vectors (yn+ ..... y)  [-Q. Q]Æ-n 
for which there exist real muber y .... , y 
i<R. 
Then So bas (s - R)-diut«#sional uteasure satisfging 

where the implicit constant 
entries of .4. 

l's-n(So) >>A 
l/ïnogradov's notation depends on s and R and the 

Proof. We apply the lemma of Nadesalingam and Pitman to the R linear forms 
Ml(y) .... ,]ln(Y) defined by ]l,(y)= Al/nLi(y) for 1 <_ i _< R, in order to 
relax the requirement A1 _> 1 of their lemnm. We note that there is a slight 
difference between the definition of [[All that ve use and the definition they use, 
which accounts for the change in the condition (8.29). Here xve have also ilnplicitly 
used the last equation on page 704 of [15] to show that the terln H(L) in the lelnnm 
of Nadesalingain and Pitman is positive. [] 

Now we return to the proof of Lemma 8.7 and apply Lemma 8.8. By (8.26) and 
the assumption T _> 1, we mav apply the lelnlna, with the choice Q = (SP )/2. 

for any w = ('1, t/-'2 ..... wR) with 
(8.30) 
lnill (1,/NI/R ) 
3T 

We obtain 

forl<i<r, and I wil< 

rein (1" Al/n) for 

3 

r+l<i<R. 

p,Æ-n(So(W, .... wh)) »  pa(,-n). 

Now, for anv choice of (Yn+, Yn+_ .... , Ys)  0(iI'l, .-- , wR), there exist real llUm- 
bers Yl,...,Yn e [-Q, Q] with Li(y) =wi for 1 < i < R. Defining z as iii (3.6), ,ve 
h ave 
Li(Pez+y) =wi for 1 <i</ï'. 
By (3.6) and our choice of Q. we also have 

P z + y  [  P , I + a P ] C [  P P  ] 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2711 

Recalling the defilfiti(m of SP, T, we sec that we have 
lts (SP, T) >>F, T-"P 
Cmbining with (8.27) and (8.28), we see Ihat Ihere is a positive rem constant 
cl = ci(F, k, R, r, s, 5, z) such that one has 
(8.31) :T(P) >_ Cl Ps-Rk. 
This completes the proof of Lelmna 8.7. [] 
Combining Lemmas 8.6 and 8.7 yields the following lower bound for 2-(P). 
Lemma 8.9. Suppose that we are in the setti9 of Theorern 1. ad that the 
coefficient ruatriz A of the systeru F satisfies (3.2) and (3.3), and that there is a 
real vector z satisfyin9 (3.6). Assume also that we bave IIFII > 1. Define if(P) as 
in (8.6). Then there is a constant ca = ca(F, h, R, r, s, 6, z) such that for P > ca, 
ont? bas 
Z(P) » ps-Rk. 
Here the iruplicit constater m l ïnogradov's ot«tion may depend on F, k,s. 
and the special real vector z. but t does hot depend on P. 
8.4. Completion of the treatment of the major arcs. We wrap up our work 
on the major arcs with the following lemma. 
Lemma 8.10. Suppose that we are in the setting of Theorem 1. and that the 
coefficient matrix A of the system F satisfies (3.2) and (3.3), and that there s a 
real v«ctor z satsfymg (3.6). Assume also that we bave IIFI[ _> 1. Then there are 
constants c4 and c, whzch may depend on F, k, s, R, r,  and z, but which do hot 
depend on P, such that for real numbers P satisfymg P >_ ca, one bas 
s 
f. 1-[ -(-/ 1-[ K(,)d. _> cP-. 
j=l i=r+l 

Pro@ Choose e = 1/(2k) and apply Lemma 8.2. Since we have g _> 2h + 1 and by 
the definition (4.10) of B, we obtain 
.« R 
(8.32) /.M HgJ(a) H Iç(oi)da-Z(P) « PS-nk(logP)-/(st¢(l+l)) 
j=l i=r+l 
Since condition (iv) of Theorem 1.2 holds, one has  » 1. By Lemma 8.9. there 
are constants c3 and c that do hot depend on P such that one has 
(8.33) Z(P) >_ cP -Rk for P _> c3. 
Lemma 8.10 follows from (8.32) and (8.33) and t.he bound 1,9 » 1. [] 

9. COMPLETION OF THE PROOF OF THEOREM 1.2 

In this section, we gather together all of our results in order to complete the 
proof of Theorem 1.2. 
We recall that we demonstrated in Section 3 that we may assume that we have 
e = 1, that we have ]IFII _> 1, that the coefficient matrix A of the system F satisfies 
(3.2) and (3.3), and that there is a real vector z satisfying (3.6). 



2712 EHIC FREEMAN 

\Ve first observe how one proves the last sentence of Theorem 1.2. namely that 
and we define m0 (,'.'.) as in (1.4) a,,d assume that we bave 
if 
have 
we 
1. 
HI 0 
8.4, noting that we «ertainly have {  2k + 1 for a suciently large «hoi«e of the 
constant C. For k = 2, we may apply Lemma 8.5. Thus we have  >>F 1. 
Now we turn to the central result of Theorem 1.2. ecall from (4.9) that we 
have 
s R 
(p)  ,.  g)(o)  Iç()da, 
(9.1) 
' J--I i=r+l 
wllere (P) was dcfincd to bc thc numbcr of solutions of the svstem (4.1) with 
.r  M (P, P) for 1  j  s. 
We first choose a flmcti(m T(P) as in Lcmma 5.5. Xç can now treat the minor 
arcs and trivial arcs. Bv Lemmas 6.3 and 7.1, one obtains 
s R 
X now «onsider the major arcs. Bv Lemma 8.10, we have 
" j=l i=r+l 
for P  ca, where ca and cs are constants that do hot depend on P. Together with 
(9.2), it follows for su«iently large P that one has 
 C5 ps--Rk 
By (9.1), for suciently large P, we have 
This establishes Theorem 1.2. 
As a final observation, we note that we llave obtained a lower bound of the 
expe«ted order of nmgnitude for the number of solutions of out svstem in a box 
of size P, for ail su«ient.ly large positive P. Recall that we assumed that we 
bave e = 1, that we bave I[Fll  1, that the coe«ient matrix A of the system F 
satisfies (3.2) and (3.3), md that tllere is a real vector z satisfying (3.6). Using 
standard t.e«hlfiques, one can check that under the conditions of eitller Theorem 1.1 
or Theorenl 1.2, without mly of these simplifying assumptions, the sanie lower bound 
holds for suciently large P. X note that in this case, P nmst be suciently large 
also in erms of e, and the implicit constant in the lower bound for (P) depends 
Oll . 

PtEFERENCES 

1. M. Aigner, Combiïatorial theor9, Grundlehren der Mathematischen Wissenschaften, Springer- 
Verlag, New York/Heidelberg/Berlin, 1979. MH 80h:05002 
2. H. C. Baker, Diophaïtiïe iïequalities, London Mathematical Society Monographs. New Series. 
1. The Clarendon Press. Oxford University Press, New York, 1986. MI 88fi11021 



SYSTEMS OF DIAGONAL DIOPHANTINE INEQUALITIES 2713 

3. V. BenIkus and F. G6tze, Lattice point problems and distribution of values of quadratic forms, 
Ann. of Math. (2) 150 (1(.)!}9), no. 3, 977 1027. MR 2001b:11087 
4. B..I. Birch and H. Davcnport, Indejïnite quadratic forms in many variables, Mathematika, 5 
(1958), 8-I2. lkII-I 20:31()4 
5..I. Br/idern and R. J. Cook, On simultaneous diagonal equations and inequalities Acta Arith. 
62 (1992), 125-149. lkIR 93h:11036 
6. H. Davenport. Analytic methods for DiopImntine eqtations and Diophantine ineqvalities, Ann 
Arbor Publishers, Am Arbor, MI, 1963. MR 28:3(102 
7. H. Davenport and D. J. Lewis, Simultaneous equations of additive type, Phiios. Trans. Roy. 
Soc. London Ser. A 264 (1969), 557-595. lklR 39:6848 
8. J. Edmonds, Minimum partition of a matroid into independent subsets, .I. Res. Nat. Bureau 
Standards 69B (1965), 67-72. MR 32:7441 
9. D. E. Freeman, A note on one cubic Diophantine inequality, .I. London MaIh. Soc. (2), 61 
(2000), no.1, 25 35. MR 2001c:11{43 
10. __. Quadratic Diophantine inequalities, .I. Number Theory, 89 (2001). no.2. 268 307. 
11. __, Asymptotic lower bounds for Diophantine inequalities, to appear in lkIathcmatika. 
12. __, Asymptotic lower bounds and formulas for Diophatine inequalities. o appear in the 
Proceedings of the Millenuial Conference in Nulnbcr Theory. 
13. L. Low..1. PiIman and A. V'olff, Simlianeous diagonal congruences, .I. Number Thcory 29 
(1988). 31-59. lkIl 89g:11030 
14. I. D. Mcir, Simultaneous diagonal p-adic equations, MathcmaIika 45 (1.(198). 337-3t9. lkI[ 
2000k: 11052 
15. T. Nadesalingam and .I. Pitman, Bouds for solutions of simultaneous diagonal equations of 
odd degree, Théorie des nombres (Québec, PQ, 1987). 703-734, de Gruvter, Berlin, 1989. MR 
91f:11021 
16. __, Simultaneous diagonal inequalities of odd degree, .I. Reine Angew. MaIh.. 394 (198.(}), 
118-158. IIR 91c:11019 
17. V. M. SchmidI. Diophantine inequalzties for forms of odd degree, Ado. Math. 38 (1980). 
128-151. lkIR 82h:10033 
18. __, Sinultaneous rational zeros of quadratic forms, Seminar Delange-PisoI-Poitou 1981. 
Progress in Math., Vol. 22, Birkhiuser, Boston, MA, 1982, 281 307. hIR 84g:10041 
19. R. C. Vaughan. On Waring's problem for cubes, J. Reine Angew. MaI h. 365 (1986). 122-170. 
hIR 87j:11103 
20. __, On Waring's problem for sm.aller exponents. H, Mathematika 33 (1986), 6-22. MR 
87j:11104 
21. __, The Hardy-Littlewood method, 2nd ed., Carnbridge Tracts in Mathematics 125. Cam- 
bridge University Press, Cambridge. U.K., 1997. MR 98a:11133 
22. Y. \Vang, Diophantine equations and inequalities in algebraic number jïelds, Springer-Verlag, 
Berlin/Heideiberg/New York, 199l. hlR 92a:11036 
23. T. D. Vboley, New estimates for smooth tl/eyl sums, J. London Math. Soc. (2) 51 (1995), 
1-13. NIR 96e:11109 

DEPARTMENT OF IATHEMATICS, UNIVERSIT'. OF ('OLORADO. 395 UCB. BOULDER, ('OLORADO 
80309 
Current address: School of MaIhemaIics. lnstitute for Advanced SIudv, 1 Einstein Drive. 
Princeton. NJ 08540 
E-mail address: freem@ias.edu 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2715-2732 
S 0002-9947(03)03200-8 
Article electronically published on Match 19, 2003 

ON THE CANONICAL RINGS OF COVERS 
OF SURFACES OF MINIMAL DEGREE 

FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA 

ABSTRACT. In one of the main resuIts of this paper, we find the degrees of the 
generators of the canonical ring of a regular algebraic surface X of general type 
defined over a field of characteristic 0, under the hypothesis that the canonical 
divisor of X determines a morphism ç from X to a surface of minimal degree 
}'. As a corollary of our results and results of Ciliberto and Green, we obtain a 
necessary and suflïicient condition for the canonical ring of X to be generated in 
degree less than or equal to 2. We construct new examples of surfaces satisfying 
the hypothesis of our theorem and prove results which show that many a priori 
plausible examples cannot exist. Our inethods are to exploit the Oy-algebra 
structure on ç,Ox. These methods have other applications, including those on 
Calabi-Yau threefolds. We prove new results on homogeneous rings associated 
to a polarized Calabi-Yau threefold and also prove some existence theorems for 
Clabi-Yau covers of threefolds of minimal degree. These bave consequences 
towards constructing new examples of Calabi-Yau threefolds. 

INTRODUCTION 

The canonical models of surfaces of general type have attracted the attention 
of lnanv geolneters. The questions on projective normality and ring generators 
of the canonical ring are of particular interest. Kodaira [Kod] first proved that 
[K.ç ' ] embeds a lnilfilnal surface of general type X as a projectivdy norlnal varietv 
for ail m _> 8. This was later improved by Bombieri [Bo], who proved the saine 
result if m _> 6, and by Ciliberto [Ci], who lowered the bound to m _> 5. We 
proved in [GP1] more general results on projective normality and higher syzygies 
for adjunction bundles for an algebraic surface. As a corollary of these results 
we recovered and ilnproved the results of Bombieri and Ciliberto on projective 
normality, and extended them to higher syzygies. We also recovered and extended 
the results of Reid [RI on the ring generators of the canonical ring of a surface of 
general type. 
An important class of minimal surfaces of general type comprises hose whose 
canonical divisor is base-point-free. Surfaces with base-point-free canonical divisor 
fall naturally into two categories corresponding to the division of curves of genus 
greater than one into hyperelliptic and non-hyperelliptic: hose whose canonical 

Received by the editors July 5, 2002. 
2000 Mathematics Sub2ect Classification. Primary 14J29. 
The first author was partiMly supported by MCT project number BFM2000-0621 and by 
UCM project number PR52/00-8862. The second author was partially supported by the General 
Research Fund of the University of Kansas at Lawrence. The tïrst author is grateful for the 
hospitality of the Department of Mathematics of the University of Kansas at Lawrence. 

)2003 American Mathematical So¢ietv 



2716 F1RANCISCO JAVIE1R GALLEGO AND BANGE1RE P. PURNAP1RA.INA 

lnorphisln maps onto a sm'fa.ce of lninilnal degree a.nd those whose canonica] mor- 
phism does hot nlap to such a surface. By a surface of minilnal degree we mean a 
nondegeuerate surface iu projective space whose degree is equal to its codimension 
pins 1. The surfaces of minimal degree are classically knowu: they are (linear) P2. 
the Verouese surface in pa, and smooth rational scrolls or coues over a rational nor- 
mal curve (see [EH]). Note that, even though we are drawing an analogy between 
hyperelliptic curves and surfaces of general type whose canonical morphisin inaps 
outo a surface of miuima.1 degree, the theory is nmch harder for surfaces, because 
higher degree covers are involved. Examples of such covers are shown in Section 3. 
In this paper we deal with surfaces of general type whose canonical morphism 
maps onto a surface of miuimal degree. The surfaces whose canonical morphism 
does hot mal» onto a surfitce of minimal degree have been studied bv Ciliberto aud 
Green (sec [('i] and [G]). (',reen and Çiliberto proved the following beautiflfl result 
regarding the geuerators of the cauouical ring: 
Let .ç bc a r«gtlar su':face of gcnc'ral t.pc witt basc-point-free canonical divisor. 
.4ssune that the caonical ltO't]istt ç satisfies the following coditions: 
(1) ç does ot ,nap X generic«dlg 2 : 1 onto the projecti,e plane: 
() ç(A-) is ,wt a surface of mi,smal degr«e (other tfian liwar P). 
Thon thc «am,i«al rig of X is gcnerated in dcgre Icss than or cqual to 2. 
The smfaces of general type X whose cauonical morphism ç maps X outo a 
others, where they play a central role in the classification of surfaces of general type 
with small cî aud in qucstious about degenerations and the moduli of surfaces of 
general type. The studv of these sm'faces h a direct bearing on the studv of linear 
series on threefotds such as Calabi-h%u threefolds, as the resnlts in [OP], and the 
authors" results in [GP2] and Sectiou 4 of this article show. 
The stmty of the ca.nouical rings of these surfaces is carried out in Section 2. 
determine the precise degrees of the generators of its canouical ring (see Theorem 
2.1). The answer depends on the degree of ç and the degree of Y. As a corollary 
of out result and the result of Çiliberto and Green (see Corollary 2.8). we fiud that 
conditious (1) aud (2) above characterize the regular surfaces of geueral type with 
base-poiut-fi'ee cauonical lmndle whose cauonical ring is generated iu degree less 
than or equal to 2. This result is surprisiug, because it contrasts with the situation 
for higher-dimeusional varieties, which, as ve show in [GP3], differs Dom the sit- 
uatiou for smfaces, ludeed, ve show lu [GP3] that there is uo higher-dimensional 
aualogue of Corollary 2.5, aud therefore there is no converse of Green's result (cf. 
[G], Theorem 3.9.3) for higher dimeusioual varieties of geueral type. 
Iu Section 2 we explain how to use the Oy-algebra structme ou ç.Ox to fiud 
the multiplicative structure of the canonical ring of X. Even though we reduce 
the problem Dom a complicated variety to a simpler variety.  a surface Y of 
minimal degree is, there are ceïtain diculties that arise in the process. The 
proof of Theorem 2.1 involves the study of multiplication of global sections of line 
bundles on a smface X of general type. To do so we reduce the problem to 
and this amonnts to stndying maps of muhiplication of global sections of vector 
bundles instead of line bnndles. This is the first dicnlty. Moreover. the relation 
between the mnltiplication maps of global sectious of line buudles on X aud the 
multiplication maps of global sections of vector bundles on  is goverued bv the 



CANONICAL RINGS OF COVERS OF SURFACES OF MINIIvIAL DEG1REE 2717 

O--algebra structure on ,Ox. However, we prove our results for a large class of 
surfaces X. Se their canonical lnorphisln g) niight, and a«tually does, correspond te 
niany, quite diverse Oy-algebra structures. This is in sharp «ontrast te hyperelliptic 
«urves, where canoni«al nmri)hisms correspond te degree 2 algebra structures, which 
are ail quite similar aud very easv te des«ribe. The algebra structures arising 
frein canonical niorphisms of surfaces are niuch lnore «ompli«ated and liarder te 
determine. This is the second difficulty one encounters, a diffi«ulty that we are able 
te overcome in the «ontext of this Imper. 
In Section 3 we «onstruct new exaniples of surfaces of general type mapping te 
a surface of niinimal degree and also re«all seine known eues. It is interesting te 
know in general what positive integers o««ur as degrees of the canonical morphisni 
if the image is a surface of minimal degree. An answer te this question is helpflfl 
in finding new exmnples. Having this I)hilosoI)hy in nfind, we prove resuhs which 
show that seine natural wavs te «onstruct examples de net work. For instance, il, 
follows ff'oto the results in Section 3 that odd degree «overs of smooth scrolls or 
«vcli« covers of degree bigger than 3 of surfaces of mininml degree, indu«ed bv the 
canoni«al morlhisni, de net exist 
The results on surfaces of general type mentioned above have ramifi«ations for 
Calabi-Ymt threefolds. In Sectkm 4 of this article we apply these results te obtain 
new results for a polarized Calabi-Yau threetbld (X, B) with B a base-point-free 
and ample diviser. Aniong other things, we find out the degrees of the generators of 
the homogeneous ring asso«iated te B, and we give a chara«terizatiou of polarized 
Calabi-Yau threefolds (X. B) whose asso«iated holnogeneous riug is generated in 
degree less than or equal te 2. The construction of examples of Calabi-Yau three- 
folds has evoked interest in recent vears. One of the iniportant sources of these 
examples is te take «overs of threefolds of minimal degree. In Section 4. we prove 
some existence theorems for Calabi-Yau «overs of threefolds of miuimal degree. For 
instance, frein these results, whi«h are more general, it follows that a Calabi-Yau 
cover of prime degree greater than 3 induced by a complete linear series canner 
corne frein a group action. 
We will expand on these ideas in two forthçoming articles, [GP3] and [OP5]. 
In the first we study the canoui«al ring of higher-diniensional varieties of general 
type whose canoni«al morphisln maps ente a va.riety of minilnal degree. One of 
the results in [GP3] shows that the converse of the theoreln of Ciliberto and Green 
for surfaces proved in this article is false for higher-dilnensional varieties of general 
type. In the second we «arry out a detailed study of homogeneous rings associated 
te line bundles on trigonal «urves. 

1. PRELIMINARIES 

Convention. Tbroughout tbis article we will work over an algebraically closed field 
of characteristic O. 

[11 this section we will recall seine known fa«ts about the push-forward of the 
structure sheaf of a variety by a fiat, finite morphisni. We summarize these fa«ts 
below, and refer for the proof te [HM], Section 2. 
Let X and 1" be algebraic varieties. Let 7r : X  Y be a finite, fiat morphism 
of degree n. \Ve have the following facts: 
1.1. The sheaf Tf.ex is a rank n, locally flee sheaf ou 1- of algebras over Or. 



2718 FRANCISCO 3AVIER GALLEGO AND BANGERE P. PURNAPRAJNA 

1.2. There exists a nlap 
1 
-tr" 7r.Ox  Oy 
of sheaves of Oy-modules defined locally as follows: Given a E rc.Ox, we consider 
the honolnorphisn of Oy-lnodules 
.Ox  .Ox 
indu('ed by umltiplication bv . Then we define tr(a) as the trace of such a 
homomorphisln divided by n. 
1.3. £tr is surjective; in fact, thc lnap Oy  u.Ox induced bv u is a section of 
2tr. Therefore the sequence 
t 
0E.Ox O 0 
split.s. E is the kernel of %tr and locallv consists of the trace 0 elements of 
 will call E the trace-zero module of u. 
1.4. u.(Ox) is a sheaf of Oy-algebras; therefore, it h a multiplicative structure. 
It.s multiplication nmp is an O -linear map 
[O-eE][O-e] O,-eE 
ruade of four components. The first component 
is given bv the multiplication in Oy, and therefore goes to Oy. The components 
OrNE O)-oE, 
E NO)- 
are given by the le[t and right module structure of E over Oy. and therefore go to 
£. Finally, there is a fourth component 
which factors through 
S2 E  0- • E , 
for nmltiplication in .Oy is conmmtative. 

2. COVERS OF SURFACES OF MINIMAL DEGREE 
Out purpose in this section is to study the generators of the canolfical ring of 
certain surfaces of general type. Specifically, we are interested in studying those 
regular surfaces of general t.vpe whose canouical divisor is base-point-free and such 
that the ilnage of the canonical morphism is a variety of minimal degree. We obtain 
the following result. 
Theorem 2.1. Let S be a regular surface of general type witl af worst caonical 
singularities and uch tlat its canonical bundle Ks is base-point-free. Let ç be the 
canonical morphism of S. Let n be the degree of ç and assume that tle image of  
is a surface of minimal degree r. Then: 
1) if n = 2 and r = 1 (i.e., if  is generically 2 : 1 onto p2), then the canonical 
ring of S is generated by its part of degree 1 and one generator in degree 4: 
2) if n  2 or r  1, then the canonical ring of S is generated by its part of 
degree 1. r(n - 2) generators in degree 2 and r - 1 generators m degree 3. 



CANONICAL IRINGS OF COVERS OF SURFACES OF MINIMAL DEGREE 2719 

The knowledge of how manv linearly independent generators are needed in each 
degree is obtained from the knowledge of the image of the multiplication maps of 
global sections of powers of the canolfical bundle. We study those multiplication 
lnaps by studying similar lnaps of a curve C in IKs]. Thus we will first prove the 
following proposition. 

Proposition 2.2. Let C be a smooth, irreducible curve. Let 0 be a base-poi77t-free 
line buTTdle o C such that 0 °2 = Kv. Let re be tbe rnorpbism induced bg I01, let 
 be the degree of rr. and assume that re(C) is a ratioal normal curve of degree r. 
Let ff(s, t) be the multiplication map 
H°(O ®) ® H°(O 'st)  H°(O'r-+t), for all s, t > 0. 
The codimension of the image of /3(s, t) i H°(O ®s+t) is as follows: 
a) If r = 1. the codimension is: 
a.1) n- 2. for s= t = l, 
a.2) 0, for s = 2, t = 1. i.e., 13(2, 1) surjects, 
a.3) 1, fors=3, t= 1, 
a.4) 1, fors=t=2, n = 2 and O if n >2. 
a.5) 0, for s >_ 4, t = 1, i.e., p(s, 1) surjects for all s >_ 4. 
b) If r > 1. the codirnensio is: 
b.1) r(n - 2). for s = t = 1, 
b.2) r-l. fors=2, t=l, 
b.3) 0. for s > 3, t = 1. i.e., ff(s, 1) surj«cts for all s > 3. 
Moreover, if r= 1 and n = 2. then the image of/3(2, 2) and the image of/3(3, 1) 
are equal. 

In order to prove Proposition 2.2, we will use the following. 
Lemnla 2.3. Let C. 0 a'nd re be as in the staterneTt of Propositio 2.2. Then 
rc.Oc = Op, ( (n -- 2)O1-, (-r -- 1) ( Op, (-2r -- 2) . 
Pro@ Since the ilnage of rr is Slnooth and of dilnension 1, rr is fiat. Then îr.Oc = 
Op @ E as Op,-lnodules, with E a vector bundle over P of tank n - 1. We noxv 
show that 
e = (n- 2)Ol:,,(-r- )¢, Ol:,,(-2r- 2). 
We have rr.0 = rr.Oc x Op,(r) and rr.tçc = rc.Oc ® Op,(2r), bv the projection 
formula. Any vector buudle over P splits: hence 
g.O C = Op (D E = Op (90p (al) -'- @ (._9pa (an-l) • 
for solne negative integers al,..., an-1 (C is connected). Then hi(Içc) = 1 ilnplies 
that exactly one of the ai's, let us say an-i, satisfies an-1 ÷ 2r = -2. On the 
other hand, since re is induced by the complete linear series ]0[, h°(O) = r + 1 = 
h°(Op (r)); so ai + r _< -1 for ail 1 < i < n - 2. Filmlly, since the degree of 0 is 
g(C)-l, we have bi(0) = h°(O) = r+l. Sillce hl((,Qp (-r-2)) --- r+l, ai+r >_ -1 
for ail 1 < i < n - 2, and so ai ÷ r = --1 for ail 1 < i < n - 2. [] 

(2.4) Proof of Proposition 2.2. In Lellmm 2.3, we have COlnpletely deternfined 
the structure of r.Oc as an Opt-lnodule. Now we look at the structure of rr, Oc 



2720 FRANCISCO .lA\ IER GALLEGO AND BANGERE P. PURNAPRA,JNA 

as an Op,-algebra. If ,t = 2. it is completely detelmined by the t)ranch diviser of 
7r on P. since in this case rr is cyclic. If 7t > 2, we observe the following: 
For seine 1 < i.j < 7 - 2. the projection of the map 
(2.4.1) Op1 (ai) @ Op, ((tj)  7r.O C te Op, (-2r - 2) 
is surjective: in fact. it is an isolnorphism. 
This is se because otherwise Op1 .) Op1 ((/1) t "'" Ç' Op1 (Ort--2) would be an 
integra.1 subalgebra of rc.Oc, free over Op of rank , - 1. Then n - 1 should divide 
7. which is net possible if 7 > 2. 
New we will use ont knowledge of Tf.ex- te stndv the lnaps/3(s, r) which appear 
in the statelnent of the proposition. We will write /4 iii place of /3(s. 1). Let 
R = H(O). Then, since 0 = Tf*Op, (r). bv the projection formula. 
RI = H0(OpI (r)). 
nl ---- H°(Op, (lr))  (, - 2)H°(Op, ((1 - l)r - 1)) ( H°(Op, ((l - 2)r - 2)), 
/1+1 = H°(OP l (Il n u 1)r)) @ (,, - 2)H°(Op, (lr - I)) @ H°(Opl ((l -- 1)r - 2)). 
Therefore an element of Rt. i.e., a global section of H°(O"l), is a sure of n com- 
I)onents, one in each piece of the al)ove decoml»osition of Rt. On the other hand, 
the product of an elelnent of Rt belongilg te one of the blocks with an elelnent of 
R is determined bv the ring structure of (gp and bv the module structure of 
More precisely, the lestriction of/3t te H°(Op (If)) ® H°(Op (r)) lnaps, in fact 
isolnorphically, ente tt°(Op ((! + 1)r)). The restriction of t3t te each of the blocks 
H°(Op ((l - 1)r - 1)) ® H°(Op (r)) naps te the corresponding H°(Op, (/r - 1)). 
This restriction is 0 if (l- 1)r- 1 is negative and an isomorphism otherwise. 
Likewise, the restrict.ion of pt te H°(Op((1 - 2)7" - 2)) ® H0(Op1 (r)) goes te 
H°(Op((! - 1)r - 2)), being 0 if (1 - 2)r- 2 is negative and an isomorphism 
otherwise. Theretbre it is crucial te tell which blocks of a given R are 0. \Ve have 
I = H°(OP '(r)), 
R2 = H°(Opl(2r)) ( (,, - 2)H°(Op1 (7"- l)). 
and if 1 > 3. 
nl = H°(Op,(lr))  (Tt - 2)H°(Op, ((l - 1)r - l))  H°(Op, ((I - 2)r - 2)) . 
Ail the direct smmnands appearing in the above formulae are nonzero, except 
H°(C)p,((1- 2)r- 2)) when l = 3 and r = 1 and (n- 2)H°(Op,((l- l)r- 1)) 
for all ! and all r when 7 = 2. We new deterlnine the image of fit- If 1 = 1. 
the image of/31 is H°((._gp (2r)), which has codimension (7 - 2)r in R.2. If 1 = 2. 
the image of/32 is H°(Op, (3,')) @ (7 - 2)H°(Op, (2r - 1)), which bas codimension 
r - 1 in Ha. If 1 = 3 and r _> 2 or if 1 _> 4. the image of 
surjects. All this proves a.1). a.2), a.5) and b). If r = 1. the image of/3(3.1) 
is H°(Op,(4r))  (n - 2)H°(Op(3r - 1)). which has codimension 1 in /74. This 
proves a.3). If r = 1 and n = 2. the image of/3(2.2) is H°((._gpl (4r)). which has 
codimension 1 irt /4. This proves the first claire iii a.4) alld the last sentence of 
Proposition 2.2. Finally, if n > 2. recall (sec 2.4.1) that for seine 1 _< i.j <_ 7 - 2. 
the projection of the lnap 

(._p1 (I/i) ("., (._p1 (aj)  Tf.(._ C 



CANONICAL FIINGS OF COVEIClS OF SUIClFACES OF MINIMAL DEGIClEE 2721 

to Op1 (--4) is sm:jective; in fact, it is an isolnorphism. Then. if 7 > 2, the image 
of ff(2, 2) is all R4. This proves the second part of a.4). 
Remark 2.5. Note that 0  = Içc- Then a proof of a.4), alternate to the one given 
above, can be obtained ffoto Noether's theorem and from the base-point-free pencil 
trick. The way in which Noether's theorem is related to the algebra structure of 
w.Oc is shown in [GP4], where we will give a diflrent., simple proof of this classical 
result for a general curve in M. 
From Proposition 2.2 we obtain the following. 
Corollary 2.6. Let C be a smooth curve. Let 0 be a base-point-free line bundle 
o C such that 0" ' = Kc. Let  be the mohi,sm iduced bg 0], let  be the 
degree of  and assume that n(C) is a ratioml normal curve of degree r. Let R be 
@o H°(o';)" Ten: 
1) if r = 1 ad  = 2. lhc rig R is g«,cratcd byit.s part qf degree 1 and one 
generator in degree 4: 
2) if r = 1 atd o > 2. te 'itg R is g«n«rated bg ifs part of degree 1 
gcterators in d«g've 2: 
3) if r > 1. the ring R is gcerated by its part of degree 1. r( - 2) generators 
i?t degree 2 and r- 1 gcn«rators it degree 3. 
Pro@ To know in what dcgrees we need generators, we look at the maI)s ff(s. t) 
of multiplication of sections. Preciselv the mmfl)cr of generators needcd in degrce 
1 + 1 is th(' codimensiou in R+ of the smn of th(' images of /3(1,1). 
- ., , [W]). In particular.  is gcnerated in degrce less than 
or equal to 1 if  snrjects for all k 2 l. Thus 1) folloxvs from part a) of Proposition 
2.2 and ff'oto the fact that the images of (3.1) and (2, 2) are equal. 2) follows 
likewise ff'oto part a) of Proposition 2.2 (note that in this case ff(2.2) surjects). 
Finally, 3) follows ff'oto part b) of Proposition. 
(2.ï) Proof of Theorem 2.1. The proof rests on Proposition 2.2. The idea is "to 
lift" the generators of R to the canonical ring of S. Let us define 
0 -Ns 0 
(i(s)  (ç2)  u((s.-,+,, - 
and let us denote a(s. 1) as a. As in the case of R, the images of a(s, t) will tell 
us the gencrators of each graded picce of the canonical ring of S. In fact, it will 
suce to prove the following: 
(a) If r = 1 and  = 2. a surjects for all 1  1, exccpt if 1 = 3. The images of 
aa = a(3, 1) and a(2.2) are equal and have codimension 1 in H°(K4). 
(b) If r = 1 and n > 2, a surjects for all 1  1, except if 1 = 1,3. The image of 
al bas codimension n - 2 in H°(K). The map a(2.2) is sm'jective. 
(c) If r  2, a is surjective if 1 OE 3. The image of a has codimension r( - 2) 
in H°(Bç). The image of aa has codimension r 1 in H°( 
- 
Thus we proceed to prove (a), (b), (c). Recall that I = ç(S) is an irreducible 
variety of lnilfimal degree and, in particular, normal. On the other hand, the locus 
of the points of I" with non-finite fibers bas codimension 2. Thus, using Bertini's 
theorem, we can choose a smooth curve C of IKs[ such that the restriction of the 
canonical morphism of S to C is finite (and fiat) onto a smooth rational normal 
curve of degree r. Let us denote by 0 the restriction of I@ to C Bv adjunction, 



2722 FRANCISCO JAVIER GALLEGO AND BANGERE P PURNAPRAJNA 

Kc' = 0 @z. Since fçs is base-point-ffee, so is 0. Finally, since HI(O\ 
is induced by the complete linear series [0 I, and therefore C, 0 and  satisfy the 
hypothesis of Proposition 2.2. 
ë prove first thc statements in (a), (b) and (c) regarding the maps t- Consider 
the following conmmtative diagram: 
o( -t HO(h-s)  Ho(K)  Ho(o) 
H°(K t)  H°(Os)  .... s   
o -e ,,o ( ,-et+ x  H o (Oat+x) 
H (K s ) ç >  's  
The rightmost horizontal arrows are smective because Ht(Os) = 0. bv Serre 
duality and by Kawamata-Viehweg vanishing. The left vertical arrow triviaIIy sur- 
jects. The right vertical arrow is the composition of the map H°(A 
H°(O t)  H°(O), which is surjective for all l  I again because H(Os) = O. by 
Serre duality and by Kawamata-Viehweg vanishing, and the nmp t of multiplica- 
tion of global sections on C, studied in Proposition 2.2. Then if foIlows from ching 
°t, t+  H°(O @+) nmps the image of t onto 
the diagram tlmt the mal) ,, "s 
the image of t, and that the codimension of the image of t in H°(O t+) is equal 
to the codimension of the image of t in H 
"s » This, together with Prop 
sition 2.2. a.I.a.2, a.3, a.5 and b. proves the claires in (a), (b) and (c) concerning 
the codimensions of the images of the maps t. 
Thus the only things left fo prove are the claires al)out (2.2) when 
consider the commuttive diagram 
o - Ho(z)  HO(O.») 
H°(Iç') H°(Ks)  H°(Iç2) H (h s )  
H°(Iç 3) ç > H°(Iç 4) » H°(O ) 
The rightmost horizontal arrows are surjective because H  (Os) = 0 and bv Serre 
duality, and by Içawamata-Viehweg vanishing. The left vertical arrow surjects, 
we bave ah'eady proven. The right vertical arrow is the composition of the map 
H 0 -2 0 
(Ks)H (0 )  H°(O)H°(O2), which is surjective because S is regular 
and by Serre duality, and the map (2, 2) of multiplication of global sections on 
C. Then if follows from chasing the diagram that the map 0 
H (I; s )  H°(O ) 
maps the inmge of (2.2) onto the image of ç(2.2), and that the codimension of 
the image of (2.2) in H°(O ) is equal fo the codimension of the image of (2, 2) 
in 0 
H (K s ). On the other hand, we know that the image of (2.2) and the image 
of ç3  ç(3. i) are equal of codimension 1 in H°(Oa). if r = 1 and  = 2. Thus we 
conclude that the inmges of (3, i) and (2, 2) in H°(Iç ) are also equal and of 
codimension i. Finally, if r = 1 and n > 2, ç(2.2) surjects by Proposition 2.2.aA. 
Thus we conclude that if r = 1 and n > 2, then (2, 2) surjects. 
Theorem 2.1 compIements known results on generation of the canonical ring of 
smooth, regular surfaces of general type. Ciliberto and Green (cf. [G], Theorem 
3.9.3, and ICi]) proved that, given a smooth surface of general type with h.(Os) = 
0 and h-s globally generated and ç being the canonical morphism, a sucient 
condition for the canonical ring of S fo be generated in degree less than or equal 
fo 2 is that: 



CANONICAL IRINGS OF COVERS OF SURFACES OF MINIMAL DEGREE 2723 

(1)  does hot map S generically 2 : 10lltO p2, and 
(2) ç(S) is hot a surface of minimal degree other than linear p2. 
As a corollary of the Ciliberto and Green result and of Theoreln 2.1, we obtain 
the following: 

Corollary 2.8. Let S be a smooth regular surface of general type and such that Ks 
is globally generated. Let ç be the canonical morphism of S. The canonical ring of 
S is 9enerated in degree less than or equal fo 2 if and only if 
(1) ç does hot map 5; 9enerically 2 : 1 onto p2, and 
(2) ç(S) is hot a surface of minimal degree other than linear p'2. 

3. EXAMPLES OF SURFACES OF GENERAL TYPE 

bi this section we construct some lleW exanlples of surfaces of general type that 
satisfy the hypothesis of Theorem 2.1. The easiest way one could think of producing 
examples would be to build suitable cyclic covers of sm'faces of mininlal degree. 
However, as the next proposition shows, onlv low degree cyclic covers can be induced 
bv the camnlical morphisnl of a regular surface. So we have to employ other means 
to construct these exanlples. 

Proposition 3.1. Let X be a surface of general type with af worst canonical singu- 
larities and with base-point-free canonical bundle. Assume that the complete canon- 
ical series of X restricts to a complete linear series on a general member of the 
canonical series (e.g., if X is regular). Let ç : X   be the canonical morphi.«m 
fo a surface of minimal degree. Let n be the degree of ç. Let I, be a smooth open 
set of Y whose complement bas codimension 2 and let L be a line bundle on U. 
Assume that 
(ç.Ox)lu = Ou @ L - @... @ L'='-' 

Then n = 2 or 3. 

Pro@ Let H be a gelleral hyperplane section of }" contained in U and let C be the 
inverse image of H by ç. Then C is a slnooth irreducible member of [tç.\-[ and H 
is a smooth rational normal curve. By assunlption the morphism 91c : C  H is 
induced by the complete linear series of a line bundle 0. Bv adjunction 0 a2 = tçc. 
Thus C, 0 and çlc satisfy the hypothesis of Lelmna 2.3, and 
((,lŒE).O'-, z Opl e (n -- 2)Opl (--r -- 1 ) @ Opl (--2'" -- 2) . 
On the other hand, (q)lc).Oc is equal to the restriction of q).Ox to H, and hence 
(Ic).OC : Op I L t-1 ( . . . ( L ri-n, 
where U is the restriction of L to H. The onlv way iii which (ç[c).Oc can have 
these two splittings is when n = 2 or 3. [] 

Corollary 3.2. Let X be a regular surface of general type with at worst canonical 
singularities and with base-point-ffee canonical bundle. Let  be the image of X 
by its canonical morphism X  Y. If Y is a surface of minimal degree and c2 is a 
cyclic cover, then the degree of op is 2 or 3. 
The next proposition also rules out manv possible examples of covers of odd 
degree: 



2724 FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA 

Proposition 3.3. Let X be a surfl«e of general type with at u,orst «anoni«al sin- 
gdarities whose «anoni«al divisor is base-point-free. Let ç be a morphism induced 
by a su.bseries of ]Kxl. I]' is generically finite onto a smooth s«roll } C Pv. then 
the degree of  is even. In particulor, there are hot generically finite covers of odd 
degree of smooth rational no,wal scrolls induced bg subseries of Içx. 
Proof. Let f bc a fiber of I and lct C be a scctiou of Y. Lct -d = C 2. Since Y is 
a scroll, its hyperplane section is linearly equivalent to C+ tuf, for some integer m. 
Then ç.x- = *(c + tuf). Then dcg  = (*f). (*C) = (*f). (Içx - mç*f) = 
(ç*f) • (Kx- + ç*f), which is an cven number.  
Now we constru«t some examples of regulm" minimal surfaces X whose canonical 
uiorlhism  maps onto a varietv of minimal degree, and also mention known ones 
relevant t« this Imper. 
The cases when  is a generically finite morphisui and bas degree 2 or 3 have been 
completely studicd bv Horikawa and Konno (see [H1], Theorem 1.6, [H2], Theorem 
2.3.I, [H3], Theorem 4.1 and [Kou], Lemma 2.2 and Theorem 2.3: see also Mendes 
Lopcs and Pm'dini, [IP]). As it t«lrns out, there exist generically double covers of 
linear p2, the Vç, ronese surface, smo,th rational normal scrolls S(a. b) with b  4. 
and cones over ratioual normal çurves of dcgree 2, 3 and 4 and generically triple 
covers of pe (in particular, cyclic triple covers of p2 ranfified along a sextic with 
suitable singularities) and of the cones over rational normal curves of degree 2 and 
3. Horikawa (see [H4], Theorem 2.1) also dcscribes all generically finite quadruple 
covers X  ', wherc X is a smooth, nfinimal regular sm'face, ç is the canonical 
morphism of X, and ) is linear p2. 
The examples of Horikawa and Komlo just reviewed are examples of covers of 
degrce lcss than or equal to 3 of surfaces of nfinimal degree and quadruple covers of 
P. XX now construct three new sers of exmnples of regular surfaces of general type 
that are quadruple covers of surfaces of minimal degree under the canonical mor- 
phism. These examples are 4:1 covers of smooth rational normal scrolls isomorphic 
to the Hirzebruch surfaces F0 and F, and of quadric cones in p3. 
Example 3.4. IIe constct finite quadruple covers X  -, where X is a s,nooth 
minimal regular surface of general type,  is the canonical morphism of X. and )" 
is a smooth rational scroll S(m, m). m  1. 
Let f be a fiber of one of the fibrations of P and let ff be a fiber of the other 
fil»ration. Then " is PI x PI, and it is embedded in p-+l by f + mf'[ or by 

bi and b2 satisfv the following: 
and b2 = 1. 
= nt + 1 . 

If I" is embedded by If'+ tuf I, let a, a2, bi and b2 satisfy the following: 
eitherb =l, b2=2. a=m+l anda2=l. 
or b = 2. b2 = 1, ai = 1 and a2 = m + 1 . 
For i = 1, 2, let Di be a smooth divisor linearly equivalent to 2(aif ÷ biff) such 
that D and D2 intersect at D • D2 distinct points. Those divisors exist because 
bv the choices of a. a2, b and b2, both 2(af + blff) and 2(af + b2f') are verv 
ample. Let X    be the double cover of " ranfified along D. Since D is 



CAN(ï)NICAL 1RINGS OF COVE1RS OF SU1RFACES OF MINIMAL DEG1REE 2725 

Slllooth, se is _\-r. Let D. I)e the im-erse image in 3. "r of D2 1)v tgl. Since D 2 is 
snlooth and lneets D 1 ai distinct i)oints, D is also snlooth. Let X  X r be thc 
double cover of .\' ralnified along D.. Since X  and D are 1)oth Slnooth, sois .\. 
Let ç = ¢'1 o .92. Now we will show that .\" is a regular surface of genel'al type, 
that Kx = (*O)-(1), and that ç is induced 1).v thc COlnl)lete canonical stries of 
X. First we find out thc structure of y).(,.9.\- as a module over Or. Recall that 
ç2,Ox = (gx, @ l*(gr(-a2f - bf'). Tllen 
@.Ox - I. Ox' t l. (l* OY(--o2f -- b2ff)). 
Since pl.O\', = Oy @ Ot-(-alf - bf'), then by the l)rojection fornmla we have 
ç.Ox = Or ©O-(-alf - blf')©Ov(-o2f -b2f')©O-(-(al +a2)f - (bt +b.2)f') . 
We sec now that X is regular. Rccall that Hl(Ox) = Hl(ç.Ox). Oto" «boite of 
al, 02, bi and b2 ilnplies that alf + blf' and a2.f + b_f' are both very ample; thus, 
by Kodaira vanishing, 
Hl(Oy(-alf - hlf')) : Hl(Oy(-a2f - h2f')) 
= H'(Or(-(a, + a2)f- (b, + b2)f')) = O. 
ThelL since H  (Or) also vanishes, se de H 1 (ç.Ox) al,(1 H  (C9.\-). \\i, nov COml)ute 
Kx. Since ç2 is a double cover ralnifi(d al(mg D., 

For a similar reason, 

I'\- = (2*(Içx' ® ç(O-(a_f + b2f')). 

IÇx, = yg(/Çy ( Oy(olf -t- blf')). 
Then 
Iç\- = ç*(tç-® O}-((o 1 -1- a2)f + (bi + b2)f')). 
Since/£v = Or(-2f - 2f'), it follows again froln the choi«es of al, 02, bi and b2 
that Kx = ç*Or(1). Finally, to sec that ç is indu«ed bv the «olnplete canoni«al 
linear series of X. we compute H°(Içx). We do the COlllI)lltatiOll iii the case 
Or(l) = Or(f + tuf'). The case OF(l) = Or(tuf + f') is analogons. Sin«e 
/\- = * or ( 1 ), 
H°(tçx) = H°(O-(1)) @ H°(O-((1 -al)f + (m- bi)ff)) 
@H°(Or((1 - a2)f + (m - b2)f')) 
Ç-H°(O'((1 - al - o2)f + (m - bi - b2)f')) • 
Again, by the «hoi«es of al. a2, bi and b2, the last three direct bllll18 of the above 
expression are 0. Se ç is indeed indu«ed by the complete canonical series of X. 
Example 3.5. H'e construct finite quadruple covers X  Y, u,here X is a smooth 
regular surface of general type with base-point-fcee canonical bundle. ç is the canon- 
ical morphism of X. and l is a smooth rational scroll S(m - 1, m), m >_ 2. 
Let C0 be the lnininml section of F1 and let. f be one of the fibers. Then l" is F1, 
and it. is embedded in p2, by [Ç0 + mfl- Let al, a2, bi and b2 satisfy the following: 
eitheral =l,a2=2, bl=m+l andb2=2, 
or al=2, o2=l,bl =2andb2=m+l . 
For i = 1, 2, let Di |)e a slnooth diviser linearly equivalent te 2(aiCo + bi f), su«h 
that D1 and D_ interse«t at D1-D2 distinct points. The fa«t that such divisors exist 



2726 FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA 

follows from our choice of al, o2, b and b2, which irai)lies that of the linear systems 
of D and D2. one is very ample, and the other is base-point-free. Let X'  " 
be the double cover of }" ramified along D. Since D is smooth, sois X'. Let 
D be the inverse image in X' of D2 by ç. Since D2 is smooth and meets D 
transversally, D is also smooth. Let X  X' be the double cover of X' ramified 
alongD. SinceX' andD arebothsmooth, soisX. Let ç=çoç2. Nowwe 
will show that X is a regular surface of general type, that L'x- = ç*Oy(1), and 
that ç is induced bv thc complete canonical series of X. First we find the structure 
of ç.O.x as a module over Or. Recall that ç2.0.x = O.x-, ç*O-(-a2Co - b2f). 
Then 
Ç. OX = I.OX'  çl.(l*Oy(-a2Co - b2f)) • 
Since ç.Ox, = Oy ç Oy(-aCo - bf), then bv the I)rojection formula we have 
ç.Ox = O- @ Or(-aCo - bf)  Oy(-aCo - b2f) 
 Or(-(a + a)Co - (b + b2)f) • 
Xê sec now that X is regular. Recall that H(Ox) = Hi(ç.Ox). Our choices of 
al, a2, bi and b2 inll)ly that a.( + bf and a2Ço + b2f are both be-point-free and 
big divisors: thus. by Kawanlata-Viehweg vanishiug, 
H(Oy(-aaCo - bf)) = H(Oy(-a2Co - b2f)) 
= H'(Oy(-(a + a2)Co - (b, + b2)f)) = 0 . 
Then, since H(Oy) also vanishes, so does H(ç.Ox -) and therefore H(Ox-). X 
now COHlpUte KX-. Since ç2 is a double cover ramified along D, 

tçx = 2*(tçx' ® ç(O)-(O2Co + b2f)). 

For a similar reason, 

K\-, = 2(tç ® Or(a,Co + b,f)). 

Thell 

Kx = o*(tQ-¢ Ov((a, + a2)Co + (bi + b2)f)). 
Since tçy = Or(-2Co - 3f), it follows from our choice of a, a2, b and b2 that 
Kx = ç*Oy(1). Finally, t.o see that ç is induced by the complete canonical linear 
series of X, we compute H°(Kx). Since 
H°(tQ) = H°(O(1)) ® H°(Oy((1 - al)Co + (m- bi)f)) 
(SH°(O((1 - a2)Co + (m - b2)f)) 
(SH°(OY((1 - al - a2)Co + (m - b - b2)f)) . 
Again, bv the choices of a, a2, b] and b2, the last three direct sums of the above 
expression are 0. So 

Remark 3.6. With the saine arguments, if one allows certain nfild singularities in 
D1 and D, then one tan construct examples of covers of Fo and F with at worst 
canonical singularities. 

Finally, we construct an exmnple of a quadruple cover of a siugular surface of 
minimal degree. 



CANONICAL RINGS OF COVERS OF SURFACES OF MINIMAL DEGREE 2727 

Example 3.7. 14"e construct an ca'ample of a mnooth, generically finite, quadruple 
cover X  Z of the quadric cone Z in p3, where X is a regular surface of general 
type whose catonical divisor is base-poit-free, ad y) is its canonical rnorphism. 

Let 1- = F2. Let C0 be the lninilnal se«tion of ) and let f 1)e a fiber of Y. 
Let D1 be a Slnooth divisor on 1, linearly equiva.lent to 2C0 + 6f and lneeting C 
transversally. Let D2 be a Slnooth divisor on 1", lineaïly equivalent to 3C0 + 6f and 
meeting D1 transversally. Su«h divisors D1 an(i D2 exist, be«ause 2C0 + 6f is very 
alnple and 3C0 + 6f is base-point-free. Note also that, silwe (3('o + 6f) • C0 = 0. 
C0 and D2 do hot meet. Let X'   be the double «over of Y along D1. Sin«e 
D is Slnooth, so is X'. Sin«e D1 lueets (7'o at two distiu«t points, the pullba«k C 
of C0 by 991 is  Slnooth line with self-interse«tiol -4. Let D bc the pullba«k of 
D2 by 91. Since D1 and D2 meet tl'ansversally, D. is smooth, and sin«e D2 and 
C0 do hot meet, neither do D and C D. Let L be the pullba«k of 2C0 + 3f by 9. 
Let X  X' be thc double «over of X' ahmg D U C. Sin«e D U C is Sllooth, 
so is X. Let 9 = 991 o 992. Thon 

(3.7.1) 

ç.O\- = ç.ç2.0x = ç.(O\-, @ L,*) 
= CO- e, (.gv(-Co - 3f) @ Ov(-2Co - 3f) @ Ov(-3Co - 6f) . 

Sin«e Co + 3f and 3Co + 6f are big and base-point-flee, bv Kawanmta-Viehweg 
valfishing and Serre duality, Hl(Ov(-Co -3f)) = H(Cgv(-3Co -6f)) = 0. Bv 
Serre duality, H  (C9v(-2C¢-3f)) = H 1 (C9v(-f))* = 0. Then. since H 1 (C9v) = 0. 
X is regular. Arguing as in Exalnples 3.4 and 3.5. we sec that 

(3.7.2) 

tçx = ç*(tç ® Ov(3Co + 6f)) = ç*Ov(Co + 2f) . 

Now we compute H°(Kx). Using the proje«tion fbrmula and (3.7.1) and (3.7.2), 
we obtain that 
H°(Içx) = H°(Ov(Co + 2/)) @ H°(Ov(-f)) @ H°(Ov(-Co - f)) 
(H°(Oy(-2Co- 4/)) = H°(Oy(Co + 2/)) . 

Thus the «anonical morphism of X is the composition of ç and the lnorphisln 
Y ¢- Z C pa, induced by the «omplete linear selies of C0 + 2f. Sin«e ¢ contra«ts 
C'o, the canonical norphism of X is hOt finite, but it is generically finite of degree 
4 onto Z, which is a surface of nlinilnal degree, as we wanted. 
On the other hand, if C' is the pullback of C0 by ç, then C' is a smooth line 
with self-interse«tion -2. Thus the lnorphisln ¢ o ç also fa«tors as ç' o ¢, where 

x LX 

is the lnorphisn from X to its canolfical lnodel X and 

is the canolfical morphisln of X. Thus ç' is an exalnple of a filfite, 4 • 1 canonical 
morphisln ff'oto a regular surface of general type with canonical singularities onto 
a singular surface of lnininml degree. 



2728 FRAN(2'ISCO JAVIEF CALLEGO AND BANGERE P. PURNAPRAJNA 

4. APPLICATIONS TO CALABI-YAU THREEFOLDS 
The results proved in Sectons 9 and 3 bave ramifications for Calabi-Yau three- 
fi»lds. F/ecall that if X is a Calabi-Yau threefold and B is a big and base-point-free 
divisor, a general member of ]BI is a surface of general type. Then the geometry 
and properties of surfaces of general type m'e directly related to those of Calabi-Yau 
threefolds. Concretely, the results we have obtained in Section 2 on the canonical 
ring of surfaces of general t,vpe can be 'lifted'" to achieve analogous results for 
Calabi-Yau threefohls in a way similar to the way in which our study of rings 
of curves Mlowed us to obtain results for surfaces of general type. On the other 
hand. constructing examples of ('alabi-Yau threefolds has attracted the attention 
of gcometers in recent vears. One of the important sources for these exmnples is 
1)recisely to take covers of varieties of minimal degree. Proposition 3.1. Corollary 
3.2 and Proposition 3.3 tell us features of generically finite covers of surfaces of 
minimal degree iuduced by the canonical morphism. We will see how these features 
pass on to generically finite morphisms ff'oto Calabi-Yau threefolds to threefolds of 
minimal degree, and, as a consequence, we will obtain, among other things, that 
many a priori possible examples of Calabi-Yau threefolds cmmot exist. 
We start with thc Calabi-Yau threefold aualog of Theorem 2.1: 
Theorell 4.1. Let X be a Calabi-I'ut ttreefold with al worst canonical sitt9ulari- 
ries. and let B be a big atd base-po#t-free lie buttdle on X. Let ç be the morphism 
ind'uced by IBI. Let  be the de9ree of ç. attd assnne tha.t the image of ç is a variet9 
of mitimal de9ree r. Then " 
1) If  = 2 atd r = 1 (i.e., if ç is 9enericallg 2 1 oto p3), the caonical rit 9 
of X is 9enerated b9 its part of de9ree 1 and on, e 9eerator i de9ree 4. 
2) If  7  2 or r 7  1. the cartoical rit9 of X is 9eerated bg ifs part of de9ree 1. 
r(n - 2) 9erterators in de9ree 2 attd r - 1 9enerators i, de9ree 3. 
Sketch of pro@ The proof follows the saine lines as the proof of Theorem 2.1. Let 
us define 
H°(13 ?) ® H¢'(B '-'-«) "(.____) Ho(B..:+t) . 
and denote 7(s, 1) as "fs- The images of "r(s, t) will tell us the generators of each 
graded piece of the ring tï__ 0H°(Bç»). In fact. it would suffice to prove the 
following: 
(a) If r = 1 aud , = 2, ",/ surjects for all l >_ 1, except if ! = 3. The images of 
"fa = 7(3, 1) and 7(2, 2) are equal and have codimeusiou 1 in H°(B-4). 
(b) If r = 1 and n > 2.7 surjects for all / >_ 1, except if / = 1.3. The image of 
? has codimension  - 2 iu H°(B»2). The map 7(2, 2) is surjective. 
(c) If r _> 2, 7t is surjective if 1 _> 3. The image of 'i bas codimension r(t - 2) 
iu H°(B®). The image of',/ has codimension r - 1 in H°(Ba). 
Now a suit.able h.vperplane sectiou of the image of ç is an irreducible surface t" of 
lninimal degree. It.s pullback by ç to X is a surface S of general type with at worst 
canonical singularities. Moreover, bv adjunctiou B 0 Os = Ks, and the complete 
linear series of B rest.ricts to the complete canonical linear series of S: so ols is the 
canonical morphism of S. Therefore S is uuder the hypothesis of Theorem 2.1. and 
the proof follows verbatim the steps given in 2.7, the role of S there being played 
by X here, the role of C there being played bv S here aud the role of Proposition 
2.2 there being played by Theorem 2.1 here. 



CANONICAL IINGS OF COVEIS OF SUIFACES OF MINIMAL DEGIEE 2729 

To how wha,t we mean, we outline how to find the ilnages of the maps t. 
consider the following commutative diagrmn: 
H°(B '») @ H°(Ox)  H°(B 5 ') @ H°(B)  H°(B :: .)  H°(Içs) 
H°(B TM) ç  H°(B ''+1)  H 
The rightmost horizontal arrows are smjective because H  (Ox) = 0, by Serre 
duality and by Kawmnata-Viehweg vanishing. The left vertical arrow triviallv sur- 
jects. The right vertical m'row is the composition of the map H°(B 
0 
H (Iç) @ H°(Ns), which is smjective for all I  1 again because H(Ox) = 
by Serre dualitv and bv Kawamata-Viellweg vanishing, and the map at of multi- 
plication of global sections on S, stndied in (2.7). Then it follows from chasing the 
diagram that the map H°(B c+l) 
 n çl S ) lnaps the image of ?l onto the 
image of al and that thc codimcnsion of the image of or in H°(N/+1) (which was 
equal to the codimcnsion of the image of t in H°(Ol+l)) is cqual to the codi- 
mension of 7 in H°(B:»l+l). This, together with the claires in (a), (b) and (c) 
coneerning the codilnensions of thc ilnages of the maps at proved in (2.7), givcs 
the codilnension of the ilnages of the lnaps 7l in H°(B l+). The clailns rcgarding 
3(2.2) are proved analogously. 
As we did in the case of the canoniçal ring of regular surfaces of gencral type, 
we can characterize when @0 H°(B') is gcnerated in degrce less than or equal 
to 2 using Theorcm 4.1 and results fronl [GP-]" 
Corollary 4.2. Let X be a Calabi- l hu thre@ld wilh at worst canouical singulari- 
ries. Let B be a bi9 a,d base-point-free line budle on X atd let ç be the 
induced by IBI. Thc @o H°(Bl) is generated in degree less than or equal to 2 
if and only if 
(1) ç does hot map X 9eterically 2 • 1 onto P:" and 
(2) ç(X) is hot a thre@ld of minimal degrce other than p3. 
Pro@ Theorem 4.1 tells us that if ç(X) is a variety of nfinimal degree, then the 
ring @lO H°(Bet) is generated in degree less than or equal to 2 if and only if 
(X) = p3 and the degree of ç is greater than 2. If ç(X) is hot a varietv of 
minimal degree, then in the prooN of [GP_], Theorems 1.4 and 1.7. it is shown that 
HÙ(B .:.' ) @ H°(B 1)  H°(B l+l) 
surjects if/ 2. 
The study of the generators of the ring @lO H°(B-l) is closely related with 
the question of when B3 is normallv generated when B is mnple. ecall that a 
line bundle L is said to be normally generated, or to satisfy property ç. if it is 
very mnple and the image of the morphism induced by LI is a projectively normal 
variety. This is equivalent to the ring @0 H°(Le) being generated in degree 
1. In the present context (X a Calabi-.u threefold and B an ample and base- 
point-free line bundle on X), the answer to the question of when B m is normallv 
generated is partially known. If the image of X by the morphism ç induced bv 
IBI is hot a variety of minimal degree, the authors gave a complete answer to this 
question for m 2 2: in this case, in [GP2], Corollm'v 1.1, Theorem 1.4 and Theorcln 



2730 FtANCISCO JAVIEI GALLEGO AND BANGERE P PUtNAPtAJNA 

1.7, they proved that B ®mis normally generated if rn _> 2. The way of proving 
those results serves to illustrate the relation between the study of the generators of 
=o H°(B®) and the nornlal generation of powers of B. For instance, in order 
to prove the llOrlnal generation of B 'v2, the first step is to show that the map 
H°(B e) ® H°(B :')  H°(B 4) 
surjects. This was pr,,ve(l in [GP2] by showing that @ï--0 H°(B»t) is generated in 
degree less thml or equal o '2 and that the mai» 
H°(/3 ®3) ® H°(/3) -- H°(B 4) 
surjects. If (X) is a vm'ietv of mininlal degree, the situation is more conlplex. If 
m > 3. the authors also gave in [GP2], Corollary 1.1 and Theorem 1.4. a complete 
answer: B' is normally generated if and onlv if m > 4 or if m >_ 3 and ç does 
not map X 2 • 1 onto linear P:t. If m : 2, mlsv«ering the question will settle the 
following 
' 
Conjecture 4.3 (cf. [(.P-I, Conjecture 1.9). Let X be a Calabi-lau threefold and 
let B bc an ample and base-poiat-free line bundle. Then B  satisfles propert!/ No 
f and onl!/ if there is a smooth non-h!/perelliptic curve Cin lB ® Osl, for some 
s • I1. 
This çonjecture would also give a characterization of when B ®2 is very ample. 
This question of v«hen B 2 is very ample is also open. 
One might ask what light the results proved in this section shed on the conjecture. 
Theorem 4.1 says that if (X) is Pa. then the ring @ï--0 H°(B®) is generated in 
degree less than or equal to 2 if and Olfly if the degree of  is greater than 2. hl 
this case, however, it was seen in the proof of Theorenl 4.1 that ,a did hot surject. 
On the other hand. if ç(X) is a variety of minimal degree different from Pa, ,a 
does surject but t0 H°(B®t) is hot generated in degree less than or equal to 
2. Therefore the strategy outlined belote to studv the normal generation of B ®2, 
which xvorked when (X) was hot of nlilfimal degree, does hot work if c2(X) is of 
minimal degree. We point out that the conjecture is nevertheless truc if (X) is 
linear Pa (sec [GP2], Corollary 1.8). This also follows from the methods of this 
article, by studying the map ,(2.2) in the proof of Theorem 4.1. Thus the onlv case 
left in order to settle the conjecture is when (X) is a variety of minimal degree 
different from Pa, which should he addressed using a subtler strategy. 
The results of Section 3 regarding the structure of the canonical morphisms of 
regular surfaces of general type onto surfaces of minimal degree has some interesting 
consequences for Clabi-Yml threefolds. As ve will see, Proposition 3.1. Crollary 
3.2 and Proposition 3.3 prevent many a priori natural examples of Calabi-Yau 
threefolds from existing. This also shows that if there do exist examples of prime 
degree Clabi-Yau covers of threefolds of minimal degree induced by complete liuear 
series, then thev cannot corne from group actions. We smmnarize this in the next 
two corollm'ies: 
Corollary 4.4. Let X be a Calabi-Yau hreefold with ai wors canonical sia9ular- 
ities. Let B be a base-poin-free line bundle. Le X  I" be the morphism induced 
bg the cornplete linear series lB[. Let n be he degree off. Let U be a smooh open 
se of I" whose complernea bas codimension 2. and let L be a liae bundle on U. 
Assue hat 



CANONICAL FIINGS OF COVEFIS OF SURFACES OF MINIMAL DEGREE 2731 

If t" is a variet.q of minimal degree, then the degree 7 = 2 or n = 3. bt partcular, 
• if  is a c.,clic cover, the degree of ç is 2 or 3 
Proof. Let t" be a suitable hyperplane section of I', and ,b' the lmllback of t" by ç. 
Then ç[s is the canonical nol-phisln of S and satisfies thc h.vpothcsis of Proposition 
3.1, and the thesis is clear. [] 
Notation 4.5. We will call a lllOli)hislll  sa.tisf.ving (*) iii the statcment of Corollary 
4.4 a quasi-cyclic cover. 
Corollary 4.6. Let X be a Calabi-lau threefold with at worst caTonical sin9tdar- 
ities. If X --, l" is a 9enericallg finite mo7»hism oTto a smooth scroll l" C pN, 
then the degree of ç is even. In particular, there are no generically finite covers of 
odd degree of sznooth rational normal scrolls. 
Pro@ The proof is analogous to that of Corolla.ry 4.4. using now Proposition 3.3. 

hl [GP2] we described what finite lnorphisms frolll ri Calabi-Ym threefold onto 
a variety of minimal degree induced bv complete linear series were possible. After 
Corollary 4.4 and ('orollary 4.6 we can obtain the following shm'per version of the 
result in [GP2] (compare with [GP2], Proposition 1.6): 
Proposition 4.7. Let X be a smooth Calabi-lau threefold, let ç be the morphism 
induced bl the complete linear series of an ample and ba.se-point-free line bundle B 
on X. and let  be the de9ree of ç. If ç(X) is a varict'g of miimal de9ree, then 
one of the .[ollowin 9 occurs: 
1) t'=P3 andn<_24. 
2) t is a smootb q.uadric hypersurface izt p4 and n = 2, 4.6.8, 10.12 or 14. 
3) t" is a sznooth ratioTtal normal scroll of dimension 3 in P5 and  = 2, 4.6, 8, 10 
or 12. 
4) Y is a sm.ootb ratioTtal normal scroll of dimension 3 i7 pN, - _> 6. n = 2. and 
X is flbered over P with a smooth K3 surface as a general fiber. The restriction of 
B to the general flber of X is hgperelliptic, witb sectional genus 2. and its complete 
linear series m.aps the fiber 2 • 1 onto a general fiber of the scroll. 
5) Y is a smooth ratioTtal normal scroll of dimension 3 in PN N >_ 6. n = 6. 
and X is fibered over p1 with a smootb abelian surface as a general fiber. The 
restriction of B to tbe general fiber of X is a (1.3) polarizatiom and its complete 
linear series maps the fiber 6 " 1 onto a general fiber of the scroll. 
6) Y is a cone over a cozic in pe. 
7) Y is a coTe over a twisted cubic in p3. 
8) Y is a cone over a Iéronese surface. 
In addition, if X bas at worst canonical singularities aTd ç is a quasi-c!Iclic 
cover, then n = 2 or 3. 

EFERENCES 

[Bo] E. Bombieri, Canonical models of surfaces of 9eneral type, Inst. Hautes Etudes Sci. Publ. 
Math. 42 (1973), 171-219. MR 47:6710 
[Ca] F. Catanese, On the moduti spaces of surfaces of general type, J. Differential Geometry 19 
(1984), 483-515. MR 86h:14031 
[Ci] C. Ciliberto, Sui grado dei generatori detl'anello di una superficie di tipo generale, Flend. 
Sera. Mat. Univ. Politec. Torino 41 (1983), 83 111. M1R 86d:14036 



2732 FRANCISCO JAVIER GALLEGO AND BANGERE P. PURNAPRAJNA 

[EH] 

[GPI] 
[a21 
[GP3] 

[GP4] 

[GP5] 
[HI] 
[H2] 
[H4] 
l'<')al 
[Non] 
[MP] 
[M] 
[OP] 
[] 

D. Eisenbud and J. Harris, On varieties of minimal degree (a centennial accourir), Algebraic 
Geornetry, Bowdoin 1985, Amer. Math. Soc. Syrnpos. in Pure and Appl. Math. 46 (1987), 
1-14. MR 89f:14042 
F. J. Gallego and B. P. Purnaprajna, Projectwe normality and syzygies of algebraic sur- 
faces, J. Reine Angew. Math. 506 (1999), 14.5-180. MIR 2000a:13023; MR 2001b:13016 
F. J. Gallego and B. P. Purnaprajna. Very ampleness and higher syzygies for Calabi-Yau 
threefolds, Math. Ann. 312 (1998), 133-149. MR 99g:14048 
F. J. Gallego and B. P. Purnaprajna, Canonical covers of varieties of minimal degree, 
Preprint math.AG/0205010. To appear in "A tribute to Seshadri--a collection of papers 
on Geometry and Representation Theory", Hindustan Book Agency (India) Ltd. 
F..I. Gallego and B. P. Purnaprajna, Some homogeneous rmgs associated to finzte mor- 
phisms, Preprint. To appear in "Advances in Algebra and Geometry'" (Hyderabad Confer- 
cnce 2001). Hindustan Book Agency (India) Ltd. 
F..I. Gallego and B. P. Purnaprajna, On tbe rings of trigonal cu'ves, in preparation. 
M. L. Green. The canonical ring of a variety of general type, Duke Math..]. 49 (1982), 
1087 1113. MR 84k:140(}6 
D. Hahn and IR. Miranda, Quadruple covers of algebraic varieties, J. Algebraic Geom. 8 
(19.q9), 1 30. MI/ 99k:14(}28 
E. llori "kawa, Algebraic surfaces of general type with small cî. I. Ann. of Math. (2) 104 
(1976), 357-387. MR 54:12789 
E. tlorikawa, Algebratc surfaces of g«neral type with small cî, II. Invent. Math. 37 (1976), 
121 155. IkIIR 57:334 
E. llorikawa, Algcbraic sttrfaces of general type with small cî, III. Invcnt. Math. 47 (1978), 
209-248. MR 80h:14012a 
E. Hori'kawa, Algebraic surfaces of general type wzth small cî. IV, Inveut. Math. 50 
(1978/79), 103-128. MR 80h:14012b 
K. Kodaira, Plurzcanonical systems on algebraic surfaces of general type, J. Math. Soc. 
.]apan 20 (1968), 170-192. MR 37:212 
K. Konno, Algebraic surfaces of general type with cî ---- 3pg - ô, Math. Ann. 290 (1991), 
77 107. MR 92i:14039 
NI. IVlendes Lopes and R. Pardini, Triple canonical surfaces of minimal degree, International 
J. Math. 11 (2000), 553-578. MR 2001h:14049 
D. Murnford, Varzeties defined by quadratzc equations, Corso CIME in Questions on Alge- 
braic Varieties, Edizioni Cremonese, Rome (19ï0), 29-100. lkIl 44:209 
K. Oguiso and T. Peternell, On polarized canonical Calabi-Yau threefolds. Math. Aire. 301 
(1995), 237 248. MR 96b:14050 
M. Reid. lnfinitesimal view of extending a hypeplane section--deformation theory and 
computer algebra, Algebraic geometry, Proceedings L'Aquila 1988. 214-286. Lecture Notes 
in Math. 1417. Springer-Verlag, Berlin. 1990. MIR 91h:14(}18 

DEPARTAMENTO DE ALGEBRA, FACULTAD DE CIENOIAS ,.IATEM,TICAS. UNIVERSIDAD COM- 
PLt TENSE DE .MADnD. 28040 lklADnD. SPAN 
E-mail address : FJavier_Gallego@mat.ucm. es 

DEPARTMENT OF .IATHEMATICS. UNIVERSIT OF KANSAS. 405 SNOW H %LL. LAWRENCE. RANSAS 
66045-2142 
E-mail address: purnaath.ukans, edu 



TRANSACTIONS OF THE 
AMERICAN MATHEIvlATICAL SOCIETY 
Volume 355, Number 7, Pages 2733-2753 
S 0002-9947(03)03272-0 
Article eIectronicaIIy published on March 19, 2003 

A CLASSIFICATION AND EXAMPLES 
OF RANK ONE CHAIN DOMAINS 

H. H. BRUNGS AND N. I. DUBROVIN 

ABSTRACT. A chain order ofa skew field D is a subring // of D so that d E D\R 
implies d -1 ( R. Such a ring R has tank one if J(R), the Jacobson radical of 
Ff, is its only nonzero completely prime ideal. We show that a railk one chain 
order of D is either invariant, in which case R corresponds to a real-valued 
valuation of D, or R is nearly simple, in which case R, J(R) and (0) are the 
only ideals of R, or R is exceptional in which case R coIltains a prime ideal Q 
that is not compleIely prime. \Ve use the group ,¥1(H) of divisorial R-i(leals 
of D with the subgroup 7-tIR) of principal R-ideals to charactcrize t hese cases. 
The exccptional case sul»(livides further inIo infinitely IIlany cases depending 
on the index k of 7-/(R) in J/(R). Using the covering group lE of SL(2, N) and 
Ihe result that the group ring TlE is einbeddahle into a skew tield for T a 
skew field, examples of tank one chain or(lers are const ructed for each possible 
exceptional case. 

INTRODUCTION 

A sul)ring R of a skew fit,ld D is called total if d in D and d llOt iii R ilnl)lies 
that the inverse d - is contained in R. It tbllows that for such rings 5' the lattice of 
right ideals as well as the latticc of left ideals is linearlv ordered bv inclusion: 5" is a 
chain dolnain. Conversely. ai13" chain donlaill  is ()re and is a total subring of its 
skew field of quotients D. The total subrings of fields are exactlv valuation rings. 
corl'eSpollding to valuation fUllctions iiito lilmarly ordered groups. In particular, if 
we take nontrivial subgroups G of the additive group (IR, +, <_) of the reals as value 
groul)s , then we obtain the COlnnmtative vahmtion rings of ralik one. Such a ring 
Cml also be characterized as a niaxinlal subring of a field, or as a valuation ring with 
exactly one llOllZ(ro prilne ideal, hl the iion-commutative case we llillSt distinguish 
between prime ideals and coinpletely prime ideals: An ideal B - 5" of a ring 5" is 
prilne if Ii I.). C_ B ilnt)lies I C B or I _C B for ideals I and I of R. If ab  B 
iniplies a  / or b  / for elements ri, b iii 5". then / is called completely prime. 
A total subring R of a skew field D will be called a chain dolnain of rank Olm if 5" 
has exactlv one llonzero COlnpletely prime ideal. This ideal will thell be J(,R), the 
Jacobson radical of R. 

Received by the editors April 10, 2002 and, in revised form. October 9, 2002. 
2000 Mathematics Subject Classification. Primarv 16L30. 16K40. 16\V60: Secondary 20F29. 
20F60. 
Key words and phrases. Exceptional chain domains, skew field, valuation, cone, covering group. 
The firsI author was supported by NSE1RC,. 
The second anthor was supported by RFBR and DFG (grant no. 98-01-04110). 

(2003 Anerican iMathematicaI Societ 3 

")73 



2734 H. H BRUNGS AND N. I. DUBROVIN 

We prove iu Theorem 1.9 that a rank one chain domain /? is either invariant, 
i.e., all one-sided ideals are two-sided, or itis nearly silnple in which case H, .l(H), 
and (0) are its only ideals, or /ï' is exceptional in which case /? coutains a prime 
ideal that is not completely prilne. The exceptioual rank one chain domains are 
classified further with the help of the group Ad(H) of divisorial R-ideals and the 
subgroup 1-/(/) of Ad(/?) of principal/?-ideals. The lattice of two-sided/?-ideals is 
then determined by the index k of 1-/(/?) in Ad(/?), and we soEv that/? is exceptional 
of type 
These results are proved in the more general case of coims P in groups G where 
a cone P of G is a subsemigroup of G so that g E G\P iInplies 9 -1 E P. 
That rank one chain doInains are either invariant, nearly siinple or exceptional 
was proved in [4]. Invariant rank one chain orders of D correspond to valuation 
fimctions from D* into (IR, +, <_). Nearl,v simple chain domains were constructed 
in [8], [16], [5] and [3]. The construction of exceptional rank one chain domains, 
however, appeared to be elusive even though Posner in [19] hinted that such riugs 
might exist, and the classification of hypercyclic rings by Osofsky in [18] is complete 
only if su«h rings do not exist. I. N. Herstein had considered the problem and this 
existence problem was also encoulitered in [14]. Ve construct in this paper excep- 
tional rank one chain domains of anv type (Ce): TheoreIn 4.4 and Crollary 4.6. 
We do this by first constructing exceptional cones Pe of type (Ce) in subgroups 
H of the universal covering group G of SL(2,1R), Theorem 3.8, and then apply 
Dubrovin's result in [11], where he constructs ail exceptional rank oIm chain ring 
of type (C1) associated with a cone 

1. CHAIN DOMAINS AND CONES 

1.1. Basic properties. A ring R is a right chain ring, if the set of all right ideals 
of R is linearly ordcred with respect to inclusion. Left chain riugs and chain rings 
are defined silnilarly. A chaiu dolnain R has a classical skew field of quotieuts D 
aud can therefore be considered as a total subring of D ([7]). 
A subsemigroup P of a group G is called a cone of G if G = P U p-1 and P 
is a pure cone if in addition P f3 p-1 = {e}. There is a close commction between 
cones Pinagroup G and right orleft orders: if Pis acone of G and a,b  
then _ < defined by a _ < b if and only if a-lb  P defines a left preorder, and a<_b 
if and only if ba -  P defines a right preorder on G. The relations "<_" and 
 ê 
are right orders and left orders on C, respectively if and only if the cone P is pure. 
Fiually, if P is pure, then the right order defined by P agrees with the left order 
defined by P if and ouly if aP = Pa for all a in C,, i.e., P is invariant under all 
imper automorphislns of C,. The group C, is then linearly ordered. 
Let P be a coue of a group G. A nolmmpty subset I of G is called a left P-ideM if 
PI c_ I and I c_ Pe for a suitable elemeut a in G. The second condition is satisfied 
for anv I ¢- C, provided I satisfies the first condition. If in addition I C P, we say 
I is a left ideal. Flight P-ideals, P-ideals and right ideals and ideals are defined 
similarly. An ideal B of P is called a prime ideal if B -¢ P and aPb C B inplies 
a  B or b E B for a, b  P. If ab  B implies a  B or b  B for the ideal B -J= P 
of P, then B is called completely prime. 
We collect elementary properties of a cone P in G. We tan assume that P -J= G. 
Let U(P) = P fil p-l, the subgroup of units of P. 



CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2735 

a): J(P) = P\U(P) is the nlaxilnal right and the nlaxilnal left ideal of P: it 
is the Jacobson radical of P and it is a colnpletely prime ideal of P. 
b): The set of right (left) P-ideals in G is linearly ordered with respect to 
inclusion. \Ve define I _< 12 if and ouly if I _D I2 for right P-ideals I and 
I» 
To sec this, one considers fil'St principal right P-ideals oP and bP in G. Then 
either a-b  P and bP C aP or b-la  P and oP C bP. If 1.2  Il, then there 
exists a in I\I and Il C oP C I., follows. 
c): There is a one-to-one corresi)ondence between the set of COlleS P' ¢ G in 
(7, that contaill a COllP P and the set of completely pl'illlP ideals B of P. 
Pro@ Let P ç P' C (7 be concs in G. Then j' ¢ J(P') and j'  P implies j,-1 ¢ p, 
a contradiction. Hence, J(P') Ç J(P) and P' = P U (P\J(P'))-. Converselv. if 
B C J(P) is a completely prime ideal in P, then P' -= P U (P\B) - is a cone 7 (7, 
in G. [] 

d): Let I be an ideal in P with I 7 P and Q = Ç]I '' 7  0. Then Q is a 
«olnpletely prime ideal. 

Pro@ If c G P\Q and ca G Q for somc o in P, then there exists n0 with c  
I'°. However, for any n there exist ai, bi G I with ca = a...a,ob.., b,,. Then 
a...a., o =cdforsonledin Pand a=db...b, G I' follows. Hcnce, aG Qand 
Q is a completely prime ideal. [] 

e): A P-ideal I will be right principal and lefl principal if and only if I = 
zP = Pz for some z ¢ G. 

Pro@ Let I = zP = Pz with zl,z  G. Then z = zla, z = bz fol some 
a, b  P. Hen«e, bza = z. Since I is an ideal, there exists b' in P with bz = zb' 
and z = zb'a follows. Therefore, b'a = 1 and a  U(P), and Pz = zP = ZloP = 
z P. [] 

f): Let P be a cone in G. The set "H(P) of all principal P-ideals of G forms 
a group with ideal multiplication as the operation. "H(P) is isomorphic to 
a subgroup of (IR, +, _<) if J(P) is the Olfl.v completely prilne ideal of P. 
Pro@ If I = zP = Pz and I = zP = Pz, then ltI2 = zPz,P = zzP and 
(zp)- - - 
-- ,, P. It follows that 7-/(P) is a group with P as identit.v. To prove the 
second statelnent let P D zP = Pz. Then Ç](zP)  =  since otherwise Ç](zP)  is 
a completely prime ideal - J(P) by d). "H(P) is therefore an ordered Archimedean 
group and the statement follows from H61der's Theorem (see [13]). [] 
g): A right P-ideal I is a principal right P-ideal if and OlflV if IJ(P) ¢- I. 
Pro@ If I = zP, then zP D IJ(P) = zJ(P). Conversely, if I is hot principal as 
a right P-ideal, then for a ¢ I there exists b G I with «P C bP,  = bj  IJ(P), 
j ¢ J(P), and IJ(P) = I. [] 
We single out cones with the property in f)" 
Definition 1.1. A cone P of a group G ha.s tank one if J(P) is the onlv completely 
prime ideal of P. 



2736 H.H. BRUNGS AND N. I. DUBROVIN 

It follows flore the definitions that a subring R of a skew field Dis total if and 
only if the semigroup R* = (R\{0},-) is a cone in the group D*. 
This relationship between a cone in a group and a chain domain is generalized 
in the next definition. 
Definition 1.2. A total subring R in a skew field D is said to be associated with 
a cone P in a group Gif the following conditions hold: 
i): G is a subgroup of D*, the multiplicative group of D. 
ii): Ever.v element d in D* tan be written as d = gal = 292 with g.g2 in 
G and Ul.«2 in U(R) so that PglP = PgaP. 
iii): RC G = P. 
We also say in this case that the cone P is associated with the chain domain R. 
Proposition 1.3. Let the total subrbtg R of the skea, field D be associated with 
the coe P of the groap G. Then: 
i): Io --> loR defitcs an isomorphimn front the lattice of right P-ideals fo the 
lattiçe of nonzero right R-ideals. The im,erse of this rnappmg assigns I C G 
lo the nonzero right R-idcal I. 
ii): The correspmMeucc dcfincd in i) preservcs the properties of being an ideal. 
a completel9 prive ideal, a prime idcal, and a principal right ideal. 

Pro@ i) If I0 is a right P-ideal. then tvo 
form a = 9u, b = g2tt2 for gi  I0 and ui  
and gl = gP. P G P follows. Therefore, a 
that loR is a right R-ideM. since gI C_ P 

nonzero elements a. b in loR have the 
U(/). We can assume that g P C_ g2P, 
-t--b = g2(ptq -t--"a.2)  /oR: this shows 
C_ R for some g  G C D. Further, if 

g  IoRçG for a 
It follows that hg  
Similarly, one can 
(I n G)R = I. 

right P-ideal Io, then 9 = 
¢I0anduCU(R) GG= 
show that I VI G is a right 

hg'u for h  L» g' Ç P and n  U(R). 
U(P); hence, 9  Io and loftG = Io. 
P-ideal if I is a right R-ideal and that 

For ii) we only show that the right P-ideal I0 is a P-ideal if and only if I0R is 
an B-ideal. Let r  R and h  Io, a P-ideal. Then r = plu for p  P,   U() 
and rh : plulh = pL'u2 tbr ulh = L'u2 with u2  LT(B) and h.k  G. Bv ii) of 
Definition 1.2 we have PhP = Pi'P; k  Io %llows and rh  Io, which shows that 
I0B is also a lefl B-module and then an B-ideM. Comrsely. if IoB is an B-ideal 
%r a right P-ideal 10, then I o = IoB  G is a P-ideal. 
Some variations of the results in this section tan be round in [12] and [6]. 
1.2. Divisorial ideals.  consider certain P-ideals for a cone P which will fornl 
a group in case P has tank one. 
Definition 1.4. Let P be a cone in a group G. The divisorial closare Ï of a right 
P-ideal I is the intersection of all principal right P-ideals containing I : 
ï: 
hPDI 
A right P-ideM I is called divisorial if Ï = I. 
If we replace the cone P by a total subring R. we obtain the definition of the 
divisorial closure of a right R-ideal and of a divisorial right R-ideM. In addition. 
we assume that a divisorial right H-ideal is nonzero. 
b collect a list of properties: 



CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2737 

Let P be a COlle in a group G, I a P-right ideal. Then: 
a): Ï_D I; 
b): I = I: 
c): gI = gÎfor any g in G: 
d): I is non-divisorial if and only if J(P) is hot a principal right ideal and there 
exists ail elelnent z in G with Î= zP and I = z,l(P). If, in addition. I is a P-ideal 
and rank P = 1, then Ï= zP = Pz and I = zJ(P) = J(P)z. 

The properties a, b, and c follow directly froln the definition. To prove d) we will 
write J instead of .l(P) and assume that ï D I aud that z C Ï\I. Then î  zP D I 
and Ï = zP follows; then I = z J, since zjP  I for some j  .I(P) leads to a 
contradiction. This also shows that .1 is hot a principal right ideM. If J is not a 
principal right ideM, then cP  zd implies z-«P  d and z-«P  P, cP  zP 
for c,z  G. This means that Ï = zP for I = zd and hence Ï D I. If zP is a 
P-ideal, then certainly zJ is a P-ideal. Conversely, if zJ is ail ideal, tllen zP is an 
ideal, since otherwise there is an a  P and a j  .l with azj = z, a contradiction. 
Finally, we asslillle that Ï # I and I is a P-ideal alld that P has rank ont'. Tllen 
ï = zP and the left order O((I) = {.q e G J gï ç ï} # G «ontains the cone P as 
well as the cone zPz - both of which are maxinlal. It fifllows that P = zPz -, 
Pz = zP = Ï and .lz = z,l = I.  
b list a property that was proved in the proof of d): 
e): I is a P-ideal if and oldy if I is a P-ideal. 
The next result shows that in the correspondence between right R-ideals and 
right P-ideals, divisorial right ideals correspond to each other if the chain domain 
R is associated with the COlle P. 

Proposition 1.5. Let R be a total subrin9 of the skew field D associated with the 
cone P irt a group G. Then the right P-ideal I is divisorial if and only if the right 

/'-ideal I B is divisorial. 
Pro@ Assume I is divisorial, 
with hP D_ I, and 

i.e., I= Ï= Ç hP. Then I/i'ç h/i'for all h cG 
hPDI 
hR. To show the reverse inclusion, let d  ÇI hB 

hPDI 
for hP D_ I and d = hr h = gin fol g  lâ. m  U(/). HellCe, 9 = hrh m-1 ¢ 
hHG = hP and 9  hP = I. d  IB follows. Now assulne that .4 is a divisorial 
right R-ideM. A =  dR. Any Stleh d = gin fbl'g G G, II G U(). Hen«e, 
dRA 
AaG = (dR)G = (gRG) = gP. which shows that 4aG is divisorial 
and A  G is nonempty, since .4 is nollzero.  

For any subset I of a group (7, we define the following three subsets of G : the 
right order O(I) = {9 e G [ 19 C I}, the left order O«(I) = {9 e G [9I ç I}, and 
the inverse I - = {9  G I IgI ç I}. 
It follows that I - = {9 e G 191 ç O,.(I)} = {9 e G I I9 ç O«(I)}. 
We have the following two properties where P is a cone in the group G : 
f): If I is a right P-ideM. then Or(I) is a COlle of G and O,.(I) is an over cone 
of P. Fnrther, I is a right O(I)-ideal and a left Or(I)-ideal. and i-1 is a 
right Or(I)-ideal and a lefl. O(I)-ideal. 



2738 H. H. BRUNGS AND N. I. DUBROVIN 

For a proof we observe that for any g in G either gI C I and g • 0(I) or I C gI 
and g-1 • Oe(I). The test of the statements follow inmmdiately. 
g): O,.(J(P)) = O«(.I(P)) = P, and j(p)2 ¢ j(p) implies that J(P) = 
zP = Pz for solne z • P. 
The first statcment follows froln Property c) iii Section 1.1 since Or(.](P)) D 
P implies that j-lj(p) Ç .I(P) for some j • .I(P). Hence, .I(P) Ç jJ(P), a 
contradiction that shows Or (J(P)) = P and similarly O/(.I(P)) = P. 
The second statenmnt follows from Propcrty g) in Section 1.1, its left S,Vlnnmtric 
version, and Property e) iii Section 1.1. [] 
Even though one can consider the groupoid of all divisorial P-ideals for a cone 
P of arbitrary rank (see also [2]), we restrict ourselves to the rank one case: 
Definition 1.6. Let P be a cone of rank one. Then .Ad(P) is the set of all divisorial 

P-ideals togetller with the operation " 
Il * I2 = Il I2 
\Ve have the following rcsult: 

* " defined by: 
foL P-ideals 11, I2. 

Theoreln 1.7. Let P be a cone of rak oe in a group G. Tben: 
 ) ./( P) is a linearl!! odered gvup: 
d) The iv«rse of an elemett I in .Ad(P) is 
) (P) is a subgro«p of.Ad(P). 
Pro@ We show first that the operation defined in Definition 1.6 is associative. 
On the set of ail P-ideals we define a relation IL  I2 if and only if Ï = Ï2; this 
is an equivalence relation. 
We are going to show next that for P-ideals Il. 1"2 the following equivalence holds: 
(+) II2 ,-, 1112. 
If I1 = Iî and I2 = Ï, thell (q-) iS trivially true. If 11  Iî, then Ï = zP D 
zJ(P) = Il and J = ,J(P) is hot right principal. Also Ïl = zP = Pz is a P-ideal 
by Property d). 
The equivalence (+) holds therefore if and only if the following equivalence holds: 
(++) JI2 ' PÏ2 = 
Hence, if JI2 = I2, we are done. Otherwise, JI2 C 12 and 12 = Pd follows for some 
d in C, by the left synlmetric version of Property g) in Section 1.1. Since I2 is an 
ideal, we have dP ç Pd, P ç d-lpd and the equalit.v d-lpd = 19 since P has rank 
one. Then dP = Pd = 12, dJ = Jd and JI 2 = Jd = dJ , dP = I2 which proves 
the equivalence (++) and hence also (+) in this case. 
Finally, we must prove (+) if I1 = Î1 and Ï D 12. Then, as above, ï2 = aP = 
Pa D aJ = Ja = I2 for sortie a in G. The equivalence (+) then holds if and only 
if the equivalence I1J -- Ï1P = 11 holds. Using the right symmetric version of 
arguments used in the proof of (++), one shows that I1J ' I. This proves (+). 
If 11 -" I and 12  I for P-ideals I, I, 12, I, then 1112  Ïl Ï2 = 1112' g'  ILI .' ' 
Hence ./bi(P) is a factor lnonoid of the nlonoid of ail P-ideals, and the operation • 
given in the definition for .Ad(P) is associative. 
Next we show that II - = P for I a P-ideM, and I-I = P follows from similar 
arguments. Since I is a P-ideM, 1-1 is a P-ideal. 



CLASSIFICATION OF RANIx ONE CHAIN DOMAINS 2739 

If I1-1 = P, we are done; otherwise I1-1 C_ J(P) = J. If I1-1 C_ Pz C J for 
some z E J, then II-lz -1 C P and I-lz -1  1-1 which ilnplies z 
P, since P has rank one. This is a contradiction since z E ,1, and 11-1 = ,1, ,1 7  Pz 
for all z E P remains as the only l»ossibility to be considered. It then t'ollows ff'oto 
Property g) that ,l 2 = J, J is hot a principal right ideal, and hence I1-1 = ,Ï= P. 
In order to complete the proof of o) and /) we show that I - is a divisorial 
P-ideal for I a P-ideal. If on the contrary, I - = zJ C zP =- 1-1 and J is hot a 
principal right ideal, then zJI C_ P b.v the definition of i-1, and l»y (+) it. follows 
that zJI C_ P = P. Since zJ = zP, we obtain z.l C_ zÎc P, and hence z E I 1 = zJ, 
a contradiction. 
This shows that .Ad(P) is a group and thal /3) holds. For 11 _D I2 in .Ad(P) 
we define I < I. and .Ad(P) then is a linearly ordered group with P as identity. 
Elelnents in ?-t(P) have the form I = zP = Pz for some z in G with zP = 
and (zP) -1 = z-lP = Pz-1; see f) in Section 1.1 and ï/) follows. This proves the 
theorem. [] 
Corollary 1.8. Let P be a cone of ratk one in a group G. Thet ,(P) and (P) 
are A rch ira edean gro ups. 
Pro@ Let B C P be a divisorial idcal. IfB C J(P) = Jor B = J ¢ J2, then 
ÇB n = 0 by Pl'operty d) in Section 1.1. If J = ,I ', we have ,I = P and henee 
Ç] B' = 0 in all cases, and B'+ C B'. Then ,+1 C_ B', since there are no further 
right ideals between B'+1 and /,+1. This illlplies Ç/' = O, and it follov, s that 
.Ad(P) and 7-{(P) are Archimedean: see also Property f) in Section 1.1. [] 
Related results can be round in [12] and [2]. 
1.3. The classification of tank one cones. The groups fld(P) and ?-/(P) will be 
used to classify tank one cones P in groups G based on the lattice of their ideals. 
In the following theorem and proof we will write J instead of J(P). 
Theorem 1.9. Let P be a cone of rank one in a group G. Then exactly one of the 
following possibilities occurs: 
A) - The cone P is Archimedean. i.e., aP = Pa for ail a in P. IVe distinguish 
two possibilities in this case: 
Al): fl//(P) = -(P)  (Z, +, _<). tt,hi«h is exactlg the case when J ¢ J Then 
every P-ideal is a power of J and the cone is called discrete. 
A2): .Ad(P)  (IR, +, _<) and ?-t(P) is a dense subgroup of ,M(P). 
B): The cone P is nearly simple; i.e., J is the only proper ideal in P. In this 
case 34(P) = (P) = {P}. 
C): The cone P is exceptional; i.e., there exists a prime ideal Q in P that is not 
completely prime. Then: 
i): There are no further ideals between J and (2. 
ii): The ideal Q is divisorial and .Ad(P) = gr {Q} is an infinite cyclic group. 
iii):  Q" = O. 
iv): There eists an integer k >_ 0 such that "H(P) = gr{Qk}. The cone P is 
said fo be of type (C) in this case 
If P is of type (C0), then 
• " D (Q)-I D...DQ - DPDJ DQDQ  DQ 3 D-.- 



2740 H. H. BRUNGS AND N. I. DUBROVIN 

is the chain of P-ideals. 
If P is of type (CI), then 
-" D Q-2 = z-2P D z-2J D (2 -1 = z-IP D z-l.] D P D J D zP 
= Q D zJ » z2P = 0 2 D z2J D ... 
is the chain of P-ideals. 
If P is of type (Ck). k > 2. then 
"'" D (Qk+I)-I D z-IPD z-lJ D (Qk-1)-I D"" 
D Q-1 D PD J D Q D Q" 
D ... D D zP D zJ 
= 0 h" D Qk+l D "'" D Q2k-1 D z2P D z'J 
= (-2 2t D Q.t+l D "'" 
is the «hain of all P-ideals. 

Pro@ If J is the olllv proper ideal of P, then P is of type/3. 
Otherwise, let Q = U I be the lllliOll of ideals of P properly contained in J. If 
.1 : = J and J D Q, then P is exceptional: for ideals I D Q and I2 D Q in P we 
ha'ce If . I2 D_ j2 = ,1 D Q and Q is a prime ideal of P. hot completely prime and 
no fmther idem exists between ,I and Q. The divisorial closure Q of Q is ail ideal 
that CallllOt be eqllal to ,1, since J would then be right i)rillcipal. Hen«e, Q = Q 
is the sma.l]est positive element iii the linearly ordered Archimedean group ,M(P), 
and .Ad(P) = gr{Q} is ail infinite cy«li« group. The subgroup (P) has therefore 
the form (P) = gr{Q t} tbr sonle k _> 0: we sav that P is of type (Ck). 
We can now des«ribe the P-ideals in ea«h case (C) if we recall (sec Property 
d) in Section 1.2) that ail ideal I is either divisorial or of the form c,l = Je with 
Ï = oP = Pe. some c ¢ G and J = J-. It will also follow fl-Olll the rest of the proof 
that if P is exceptional, then J = j2 and J D ( = U I. where the ideals I are 
properly COlltailled iii J, the prime ideal that is llOt completely prime. 
In the case (C0) there are no principal ideals ¢ P and the group ,l(P) = gr{Q} 
«ontains all P-ideals ¢ J. In the case ((71) the ideal Q = zP = Pz is principal 
and .Ad(P) = (P). In the case (C), k _> 2. the ideal Q is principal. However. 
Qh itself cmmot be principal, sin«e otherwise Q = zP implies QhJ ¢,Qh; hen«e, 
Q,I ¢ Q and Q is principal (sec Property g) in Section 1.1). Hence Qh = zP = 
Pz D Qk = zJ = Jz for an element z in P. 
It remains to consider the case where either J ¢ J, j2 
or J = and J = Q = UI 
for ideals I properly contained iii J. hl this case we will prove that aP = Pa for all 
a in P. If for some a in P the right ideal aP is hot a left ideal, then an element c 
exists iii P with cap D aP and caj = a follows for ail element j in J. Bv assmnption 
there exists an ideal I C_ J with j ¢ I and ["] I n = : we obtain the contradiction 
a = caj = cnaj   Ç]I n . We have Pa ç aP, P c_ aPa -1 and P -= aPa -1 since P 
is of rank one. Therefore, Pa = aP for all a in P and P is invariant. 
If J =/: J', then J = aP = Pa, fol" some a in P, is the smallest positive element 
in the Ar«himedean group .Ad(P). Hence, ,4(P) = (P) = gr{J} is the group of 
all P-ideals. 
If J = j2 and J = Q. then (P) is isomorphic to a dense subgroup of (IR, +, _<) 
and .Ad(P) is isomorphic to (IR, + _<). [] 
If R is a «hain order of rank one in a skew field D. then R* = R\{0} is a cone 
in the group D*. We say that R has type (A). (Al), (A). (B), (C), or (C) if and 
onlv if the cone R* is of the saine type. 



CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2741 
The next result follows fl'oln Prol)ositions 1.3 a.nd 1.5 and Theoreln 1.9. 
Corollary 1.10. Let P be a coe asso«iated witb the rank oe cbai domaiu R. 
The'n P and R bave the saine type. 
2. THE UNIVESAL COVEmNG GROUP  OF SL(2,) 
2.1. The group SL(2,). By SL(2,) we denote, as usual, the group of 2 x 2 
lnatrices with l'cal entries and deterlnilmnt equal to 1. Then 
U= = 0 a - 

and 

3 = r(t) = Sill t COS t t  ] 

are two particular subgroups of G. Every C|elm,nt s  SL(2, ]) ca.n be written in a 
Ulfiquc wa" as 
s=r(l)l, for r(t) a with 0t <2 and u . 
To prove this clailn, let {e, e2} be the standard basis of 2, the Euclidean plane, 
and let the elelnents of SL(2, ) be the lepl'eSeltatiols of linear trans%l'lnations of 
2 with respect to the basis {ci, e}. For every nonzero vector a   there exists 
a unique element t e [0, 2) with a/llall = el cos t + e2 siu t: we write arg a = t iii 

this case. 
Let t = arg S(el) for the givell elelnellt 8 • SL(2, 
SOlne elelnent v. since r(-t)s(cl) = ael for o > 0. 
representation is tllliqlle, Sillce lJ Ç/ ---- { [}. I -- (  ï ) 
2.2. The group G. We are going to constrllct the 

ll) and r(-t)s = u • IL ! fol" 
Hence, s = r(t)u and this 
• the identity of SL(2,1R). 

Ulfiversal coveling group G 

of the group SL(2,1R) in this section. \Ve do this first fOl" the subgroup  bv 
fixing a sylnbol, say x, and bv rewriting the additive group of the rem munbers 

ill lnultiplicative fOl'lll: 
R = {x t I t • ll}; X t' • a "t= = Xt'+t2: x t' 
Then R is a linearlv ordered group isolnorphic to 
R to g with r(x t) = r(t) is a group epilliorphisni with 

<_ x t2 z;îe t l _ t2. 
+, <_). The mapping r frolll 
the cyclic subgroup gr{x 2r } 

as its kernel: r is a cover of the Lie group g. Next we define the covering group G 
of SL(2, IR) as the set G = {xtu I xt • R,  • U}, the Cartesian product R × U, 
together with the following operation: If x t Ul, xt2zt2 are two elelnents in G and 
tu = 2rrk + ç for k • Z and ç • [0,2rv), then ulr(ç)u = r(b), in SL(2.1R) for 
u • U, b • [0, 2rr), and the product in G is defilmd as xtlUl -xt2u = 
The mapping r from above can be extended to a lnapping froln G to SL(2, 
by defilfing 
(") = ,-(t). 
We want to prove that G is a group and that r is an epilnorphism from G onto 
SL(2,IR). 
Lemma 2.1. The mapping r is onto SL(2, IR). and if a • b = c for elemerts o, b. c 
in G, then r(c) = r(a)r(b). 



2742 H.H. BFIUNGS AND N. I. DUB1ROVIN 

Pro@ The element a.t u in G satisfies v-(.rtu) = r(t)u for the arbitrarv element r(t)u 
in SL(2, OE); ç is onto. Ira = zUl, b = z»u in G, t = 2k+ç, k G Z, ç G [0.2) 
and if ur(ç)u = r()u.   [0, 2). ui, u  U, then c = .rt++'u and 
= (0)(2 + ç) = ()(), 
which proves the lclnlna..  

Several special cases of the associative law fol" the operation defined for G are 
proved in the next few steps. We can consider R as well as U as subgroups of G 
ad the equations 
(+) .r t . u = ad'u, .r t-'tu=.r+tu, and .rtu-u'=xtzu ' 
fifl|ow. We conç|ude also that .r t • a = .r t • b imp|ies a = b for elements a. b G G. 

Lemma 2.2. For any element g = xtu  G atd any m  Z the product g..'' is 
cqtml fo 

Pro@ We have 7rm = 2rck + 99 with/," G Z, and  = 0 if m is even, and  = 7r if 
m is odd. In both cases ur(ç) = r(ç)u follows, which proves the statelnent of the 
lemma. [] 

Lemma 2.3. For an.q a, b  G and ang integer m  Z the following equalities hold: 
.m. («. b) = (.<"- ). b = . («m. b). 

Pro@ Because of (+) the first equation follows, and we can assulne that a = tt  U 
and b = ..t ¢ R in the second equation. 
It remains to prove the following equality: 
(«m. ,,).  = . (m. ) 
where t = 2k + ç, k  Z, ç  [0, 2) and ur(ç) = r()u' for   [0.2), t  U. 
Then (.rm.u).t = zm++g'U '.  distinguish three cases in order to compute 
the right-hmd side of the above equation. 
In the first case, m = 2k' is even and the equality follows ilnlnediately. 
In the sec(md case, wm = 2wk' + w for some k' G Z and  < . Then 
tf • .T 2(k+k )++  .2(k+k )tf - 3 "+  .2(k+k')++g.gt  
since ur(g + ) = lll'(g)T() = T(g)7"()lt t : T(g + )U t in SL(2, N); the equation 
is proved in this case. 
hl the final case, m = 2k' +  for U G Z and ç  . The right-hand side of 
the above equation is then equal to 
U "  "2(k+kt+l)+-  W(k+kt+l)--+llt  
which proves the lelnlna. 

Lemn,a 2.4. Letu G U andt  (Trm, Tr(m+l)) for sornem  Z. Then u.x t = x t u 
o.' ç t'  (,.(,+ )). 

Pro@ Let t = 2rrk+ç for k G Z, q G (0,2rr). If rn = 2k is even, then  G 
(0, rr); hence sinç > 0. It follows that for any u = (0b_l)  U; the argulnent 
a cos  
/, of //r(ç)(el) = r())ttt(el) is also il, (0, rr) since b = a.rg [( 0 o, b-l ) ( SiYl(t'9 )] and 
a - sin ç > 0. Hence t' = 2rrk + ç G (Trm, 7r(m + 1)) as stated in the lelnma. 



CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2743 

If rn = 1 + 2k is odd, then t = 2rk + 
with ur(ç) = r()'u; is also negative with the above argmnent; hence,   (, 2) 
and t' = 2uk + 
Theorem 2.5. a):  is a group; 
b): The mapping r is a homomorphism frvm  onto SL(2, ); 
c): The center of G is the i¢nile cyclic group gcnerated by x . 
Pmof. To show that the operation defined for G is associative we consider three 
elements xt'ui  G, i = 1, 2, 3 with ri   and ui   and thc equation 
By Lemmas 2.2 and 2.3 this equation holds if and onlv if the following equation is 
true: 
(x"+ " *+)" x+" =/'+%" (*+%" *+a) 
for integers k, m and n. 
It follows that it is suffiçient to prove (.) only in the case where t, t2, t3  [0, ). 
For g = x«u ' and g2 = x 'u with t, t'  N, ', "   we apply Lenmm 2.1 and 
obtain 

,-(t)' = (,'(ri).1-T(t2),,2)- T(t3),3 

and 

in SL(2,1R) where the operation is associat.ive, and therefore r(t)u' = r(t')u" follows. 
This implies u' = u" and t - t' = 2rrk for sonle "  Z. It remains to show that 
W apply Lemma 2.4 ad obtai z  «z for z  U,   [0, ); z 
«ta = «t3 for   U, t3 e [0, n); u2x ta = xt3 for u 2  U, t 3  [0.); and 
l-t2+t : .çt2'3ï for uï Z ç and t2,3  [0, N). 
Therefore: 

gl = (XttUl " Xt2t2) " Xt3U3 = '{xtt+t'U'l U2] " Z3U3 

and 
g2 XttUl (3:t2t2 :Et3t3) xtltl 3;t2+t'3 t 
: • . : • 23 
t+t2 3- Il I 
= " ' "1 23" 
Hence, t = t + t + 3 and t' = ri + t2,3 and therefore 
t-t'= ' 
t2 + 3 - t2,3 = 2k. 
However, t + 3 and t2,3 both belong to [0, 2) and k = 0 and the associative law 
follows for the operation defined for . 
Since  bas e = x°E, for E = ( ï), as the identity and xtu bas u-x -t as its 
inverse,  is indeed a group; this proves a). 
The statenlent b) was proved in Lemma 2.1. It follows from Lenmla 2.3 that 
gr {x  } is contained in the center Z() of . Conversely, if xtu  Z() for t   
and u  U, then an application of Lemma 2.1 shows that r(t)u is in Z(SL(2,)). 



2744 H. H. BRUNGS AND N. I. DUBROVIN 

Ht'nce r(t)u = +r(0), u = ( ï), and t = 7rk tbr some k e Z follows. Therefore 
a-  gr{x}, which proves c) and the theorem.  
See also [1] for the fact that G is right orderable, but hot locally indicable. 
2.3. The representation of the group G. To each element 9 = xta  G we can 
assign the projection v(9) = v(x t u) =   . The nmpping I • G  Aut (, 5) is 
defined as 1(« ) = v(9 xt) for9  G, x t  R. That 1 is indeed an automorphism 
of (R, ) folh)ws from the next result. 
Lemna 2.6. For g  G let l be dçfined as above. Then: 
b): 1 is th« id«ntity mapping if and only if g is the identity element in . 
): T,« .bim« t(') = {g e   (-') = .,-'} i « o *çx -  V, 
wbich is an Ore group. 
d): 1 is an automorphism of (R, ) for «v«ry g  . 
Proof. To prove a) we comlmte v(ggx t) and ,,(ge,(g2x*)). Let g = x t' ai, g2 = 
a:t2u2 for ai  U. Then gig2x  xtux2a2xt xtuixt2x t u for some  
U, '   with u2,r t = s.t' u'. Furthcr, a.t ua.t2+t' u' = xt'+ï u' for ux t2+t' = xFù 
fl,r   U, ï  . It follows that v(gg.2x ) = a: '+ï and that v(gv(g2xt)) = 
v(xt, u.r2+t') = t,+ï; this proves a). 
To prove b), assmnc g = .t u and l(x t) = x t for all t  . For t = 0 it follows 
 and assume that u  ). Then (x) = x 
that t = O. X consider t =  = (ô  
implies that 
r[( l)(ï 1)()] :r(l): . 
, ). Finallv. for t =  we must have 
Hence. b=0and u=(ô . 
arg  = - 
and a  a-a  1 follows: hence, 9  e, the identity in , and b) follows. 
To prove c) we observe that st(x°) = {x tu, G  ] ç(z0) = wt = } equals 
U. Hence, I(x t) = xtU -*  U. These stabilizers are Ore groups in the sense that 
the group ring TU over a skew field T is an Ore domain. This is true since U is the 
semidirect product of the following two torsion free abelian groups: 
This proves c). 
Finally, we want to prove d). Since l  is an atttomorphism of (R, ), it follows 
from a) that it is enough to show that ç is an automorphism of (R, N) for any 
u  V. Ve show first that z t > z t implies I(z t) > t,Ç( ) which then implies 
that l is one-to-one and order-preserving. Bv Lemlna 2.4 and Theorem 2.5(c) we 
can assume ri, f2  [0. g). It then follows that  - f  (-, ), and in addition 
f -  > 0 if and only if 
( cos t cos t2 
sin(t-¢)= Det,sint sint) >0" 
 hv (t) = (d)  d  [0.) ,d d = g(,('))  [0.). Th, 
sin t, 
Det (u(C°s t, cos t )) > 0, since Det (u) > 0, and, as in the previous argument. 
sin t sin t2 



CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2745 

de > dl follows. This shows lhat x d2 = l;(a "t2) > ..d = lç(x h) for t2 > ri and 
that IÇ is order-preserving and one-to-one. 
It remains to show that 1 is onto, and by Lemma 2.4 and Theorem 2.5(c) 
it is enough to show that 1" maps the interval [x °,.r '] onto the interval [x °. x']. 
This, however, follows from the fact that l(a "°) = x °, l(z ') = x' and that 1" is 
continuous. [] 

We will prove next a technical result, which will be used several times. 

Lemma 2.7. Let g = ztu  G with t = 7ch + fo and x q  R with t = 7cm +/10 
for k, m  Z and fo, tlo  [0, 7c)..4ssume that (  )  ]I 2 ?.t, ith arg (  ) = tlo. 
Then l(,r h) = x rr(k+m)+t' for t' = arg (r(to)u( ; )). 

Pro@ Bv definition we have that l(a "t' ) = u(9a 31). Further, «{1 = OE.'{L-'I = 
*(k+m)zt°'tza'h° since z  is in the tenter of G by Theorcm 2.5(c). 
By Lemma 2.4 we have ,«xt° = x i  wit h  G U alld ï= arg (ur(tl0)(  )) e [0. g). 
Hence, z'u.r t'° = x'°+tù. ()n the other hand, t' = arg(r(t0)u(;)) = t0 +  
since both t0, ï  [0,). It fillows that 9.r t = a-(k+)+t°+ia and l(3 "ri) = 
.r(k +m) +t' .  

3. EXCEPTIONAL CONES IN THE UNIVERSAL COVERING GROUP 
In this section we construct exceptional cones of type (C) fol" everv k in the 
universal covering group 
We define first two particular elements Wl, tt' in G which will play m important 
role in this construction. The element Wl = ( î) e U C  and r(wt) = a'l tbllows. 
Next we consider the elemcnt ( 
(0, w) and define wœe   where u = r(-a)( ï) e U; hence, r(wœe) = ( ï). 
Lemma 3.1. Let b be an elemet in [0, u). The lira ç;»ï(z b) = x °. 
Pro@ We consider the real mmfl»er b.,, with x b = V (xb). Since u,ï = (   ) and 
r(wî) = wî, we can apply Lemma 2.7 and obtain 
bn = arg [( n ) (cosî)] = arg(cosb+2nsinb) 
sin sin b " 
If b = 0, then b = 0 for all n  0 and the result follows. If b  (0. ). then sin b > 0 
and linl (cosb 
 are now read3 to define one of the main objects of this paper: 
e = { e ç [ (.0)  0}. 
The next result shows that this is an exceptional cone of type (C 1) in 
Theorem 3.2. a): ThesetF={ge[ I(x °) Rx°}isaconein u,ithU(F)= 
U. 
b): Any right F-ideal is either a principal right ideal xtF or of the form 
xJ() for some t 
d): The cone P is exceptional of tank one with Q = «P the prie ideal that 
is hot completely prime;  is exceptional qf type (C). 



2746 H. H. BRUNGS AND N. I. DUBROVIN 

Pro@ a) If g and h are eleinents in IP, then th(:r °) = V9(Vh(:r°)) _> ç(x °) OE 0 
bv Lemina 2.6 a) and d), and gk   follows. 
If g is not in P, then lÇ(x °) < x°; hence, x ° < I'- (x °) again by Lemma 2.6. 
and g-   follows and  is a cone of G. It also follows from the above arguments 
that g, g-   implies I(x °) = x ° and g  U. Conversely, U C  and U() = U 
follows. Hen«e, J() = {g e G  I;(x °) > .r°/. 
b) Let I be any right -ideal in G. Then it follows that x  = iiff{ IÇ (x °)  g e I} 
exists since I Cc for soine c  G. çk  will show that Ï = x for the divisorial 
closure Ï of I, see Definition 1.4. By definition we have x  g = xç for ail 
g  I since a N ff; hence x  I. Conversely, if h  G with h = xT  I. then 
7  ç(x°) for ail g in I and 7  a follows; hence Ï = x. It follows that either 
I = z = ïor that Ï= x and I = x%/(P); see Property d) in Section 1.2. 
c) Assulne that zt is a -ideal. For t = m + t0, m  Z and t0  [0,) it 
follows that x° is also a -ideal since x  is central in G. If t0 > 0, it follows 
froin Leinina 3.I that there exists a power wï of w in U with wïxt° D xt°, a 
coixti'adiction that shows that xt = xm. If I = xtJ(P) is a -ideal, then Ï = xt 
is a -ideal by Property d) in Section 1.2, and t = mn by the above arguinent. 
d) We have  D J() D x = Q and Q is hot a coinpletely priine ideal of , 
since z /- x /  Q, but z /  Q. However. Q is a prime ideM, since anv ideals A 
and B of P that contain Q properly, also contain J(); hence. AB  J()J() = 
J() D Q, and it follows that Q is a prime ideal that is hot completely priine. 
There are no firther ideals between J() and Q. and  Q = . It follows that  
is an exceptional cone of type (C1): see Theoreii 1.9.  

We denote by F the subgroup gr{wl,w2} of G generated by u,1 and w2. This 
subgroup is mapped by - onto the subgroup gr { (  î), ( z  ï) } of SL(2, N) generated 
by ( î) asd (z  ï). Since this subgroup of SL(2, N) is free (see [I5], 14.2.I), the 
group F is free of tank 2. 

It follows [roln Lemma 2.7 that 
t' xarg ( -3 ) 
Iïx(x )=x = , since 
and 

,i -1 = ( ï) e u 

t'= arg [(-]z)()] = arg(). 

By a flrther application of Lemma 2.7 we obtain 
I)q(X O) = Vw2 (xarg(  3)) 
with 

_ 2F t" 

t" = arg ['r(wz)(  )] = arg [(91 ï)( - )] = arg(-). 
= x '+g(]), which proves the lemma. 

Hence, I/), (x °) = .r arg ( ) [] 



CLASSIFICATION OF IANK ONE CHAIN DOMAINS 2747 

In order to construct further cones we consider a, subglotlp H of G that contains 
F and defilm 
PH =Hn?. 
It follows imInediately that PH is closed under multiplication. If g  HPH, then 
g   and g-  6 H a  = PH follows; PH is acone of H. 
Lemma 3.3. Let t  . Then PHXt = xt. 
Proof. Itis enough to prove this for t 6 [0, ), since t = k + t0, t0 6 [0, ) in the 
general case with x  in the center of G. 
If t = 0, then PHXt =  = xt. If t 6 (0, ), then for any j 6 J() there exists 
an n with wîxt  j by Lelnlna 3.1. Hen«e, xt = J() = Uwîxt ç PHxt, 
and the statelnent in the lemlna follows. 
The next result shows that F contains eleinents of a certain type. 
Lemma 3.4. For any integer m and ay  > 0 there ests an elemet xtu in F 
with "  U ad t  (m,  + ). 
Proof. Let h be the element www in F. Thon, by Lemma 3.3, we have 
I(x +) = x ° where fl = arg(). It follows that lh(x °) < l7(x ) = x - 
and that tkç (x °) < x -for any natural number N. 
 conclude that for the given integer , there exists a natural mmber N and 
an integer AI < m with 
For e the given rem number, there exists by Lelllma . and tlle contimlity of I a 
 with 0 <  < e and 
«, ([°, x)) ç (,x+) 
and hence 
(.) , ([x«+)) ç ((+«(+1+) 
follows for all k 
Bv Lemma 3.1 there exists a natural number n with 
çÇï,h(x °) [xM,x M+5) and h=h F. 
Hence, by (*) we obtain 
By another application of Lenlllla .1, there exists a lmtural llllnlber 2 with 
By repeating the last two steps m - (M + 1) tilnes, the statement of the lelnllla 
follows. 
The next result shows that the cones P are indeed exceptiolml. 
Proposition .. Let H  F be a sbgrop of  ad P =   H. The: 
a): P is an ezceptioal rak oe cone i H. 
defiees a isomorphism betwee () ad (P). rhe iwverse 4 ç is 
9iven b ç-(C) = C for C a divisorial P-ideal. 



2748 H. H. BRUNGS AND N. I. DUBROVIN 

Pro@ We recall that AA(IF) is the group of divisorial IF-ideals in G (Definition 1.6) 
a.nd that AA(IF) = gr {Q} = gr (xN?) by Theorenls 1.9 and 3.2. 
If C is a divisorial PH-ideal in H. then C'IF is a IF-ideal in C by Lenmla 3.,-1. The 
divisorial closure C' of CIF is thereh)re equal to solne power of ,rrIF and C' = 
follows for some m in ]. çVe want fO prove that CI? ç3 H --: C and assulne that 
hPH D_ C for some h C H. Then hPHIF = hIF D CIF; hence hIF __D CIP. Therefore. 
hPH = hPçH D_ CIFç H. It follows tlmt C= C D_ CIFçH D_ C" and C" = CDH. 
This shows that Cbeing a divisorial PH-ideal ilnplies C = xIF ç3 H for some m. 
We want to show next that (z'IF-"-H) : :r''IF  H for anv u. Since 
is divisorial, we know that (a«'IF-'H) = xIF (3 H for SOlne m by the above 
al'glllll(qI. 
Bv Lemma 3. there exist elelnents x tul, xt2uz  F C H with 
N, u,,u2 G U a.nd ri,t2 G (Tf(tf - 1).Tf(t, - 1)+ ). 
It fillows that 

• rt'l PH = u.t UlIF f-? H D .r "- u2IF  H = x t u2Pu  x P  H. 
Hence, .r(n-l)P  H  (.rPH)  xP H. 
If «(-I)pH = (zPH), then this ideal would also be equal to .rtPH and 
• r t: uPH. This would ilnply xttl PHP = zP = ,rtuPH  = ztP, a contradiction 
that shows that (.r  H) = (zP H) for all n. This set of divisorial P-ideals 
does hot contain J(PH), does hot contain a completely prime ideal (Lelnm 3.3 
and 3.5) and no ideal of the folIn zJ(PH) ¢ J(PH), «  PH, is colnpletely prime 
in PH. This shows that PH has rank one and that «(PH) is infinite cyclic with 
QH = z  H as the positive generator of ,M(PH). Since J(PH) D QH. it follows 
from Theorem 1.9 that PH is a.n exceptional tank one cone in H. This proves all 
sta.telnents in the lemma.  

We consider now a condition that will guarantee that PH is exceptional of type 
(c). 

Proposition 3.6. Let H be a subgroup o.[G containing F with H (gr {x'} x U) = 
gr {x 'k} x U(PH) for some integer lr >_ O. Then the exceptional cone PH bas type 
(C). 

Pro@ It was shov«n in the previous proposition that PH is an exceptional cone 
with AA(PH) = gr{(xIF (3 H)}. To prove the statement in this proposition it must 
be shown that 7-l(PH) = gr {xPH}, sec Theorem 1.9. Hence, let gPH = PHg be 
a principal ideal in H (sec property e) iii Section 1.1). 
Then 9IF = PHgIF = IF9 ]P by Lenmm. 3.4 and gIF = IFg since IF has rank one. 
By Theoreln 3.2, c) it follows that g = .r" G H for some integer m and u c U 
and /7 = Xkr for u  U(PH) and solne integer n bv assumption. Therefore, 
gPH = xrrrpH and H(PH)= gr {xrtpH} = gr {Qk} follows for Q = xIFçH: PH 
is exceptional of type (Ck). [] 

Theorem 3.7. Let Hk = gr {u) 1, W2,.r rrk} be the subgroup ofG generated by F and 
the central element x  for an integer k >_ O. Then Pk = IF V Hk is an exceptional 
tank one cone in Hk of type (Ck). 

Pro@ It is sufficient to verifv the conditions iii Proposition 3.7 for H.. 



CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2749 

Assulne that 
(*) "]Ç/r/';P'/ïl'/-'I /J-'ï2"/-' 2"'" u'ïnw n = .r=m* • Hk R (gr {z =} x U) 
for solne integers p. ui, Pi for i = 1 ..... , and u  U. b apply the mapping 
(Theorem 2.5b)) to both sides of thc above equation and obtain 
12Wl 1 î)...(2ïn 1 a 1) 
(**) (-1)P(o 1 )(2"1 
where u = (  
0 _1) with b, 0<aGN. 
Since the entries of the lnalrices al the left side are all integers, it follows that 
and a - are int.egers greater lhan Zel'O: hence a. = a - = 1. By a silnilm" argument 
it follows that b is an even inleger, b = 2s for some s in Z and u = (  ) = (  î ) = 
wf G r(F) follows. 
If (-I)P(-1)  = -1, then it follows from (**) that 
__(î) = (2ïl)(2 1 î)...(2n)( 1 
..î)( ) e (F), 
which is a contradiction, since the group r(F) fl'eely generated bv r(u,1) and 
(see the relnarks bcforc Lclmna a.3) does hot ('ontain a nonIrivial central elelnCnt. 
Therefore, (-1)  = (-1)  can 1,e can«clled in (**) and, using again Ihe fa.ct that 
b = 2tq if we ignore expon«nls lhaI could l»c zcro. Wilh u = wï ve tan rewrite 
= z tv . Il fifllows that tq = s, m = kp and u = 
the condition in Proposition 3.7 is satisfied and Theorem 3.8 follows. 

4. EXAMPLES OF EXCEPTIONAL ANK ONE CHAIN DOMAINS 
In this section we construct domains S associated with the exceptional cones 
Pk of type (Ck) as described in Theoreln 3.8. 
In Lemma 2.6(c) it was proved that TU is an Ore domain for any skew field T 
and the subgroup U of G.  denote by K the skew field of quotients of TU for 
a given skew field T; for exalnple, T = Q, the ratiolmls. Let lç{G} be the right 
K-vector space and Ieft T-vector space consisting of all series 
with ri < t < .... k  Iç. and supp () = {a "t' [ k ¢ 0} well ordered. 
XZ call snpp () the support of the series . If k ¢ 0. lhen v() = z ri 6 R is 
the norm of 7 and v(0) =  for 7 = 0. 
Let Q = End Iç{G}- be the elldolnorl)hisln ring of the K-veclor space Ix {G}ç. 
For q G Q and  G K{G} we write q[7] for the ilnage of  under q. The representa- 
tion V : G  Aut (R, 2) considered in Section 2.3 can be extended to a mapping 
V defined on Q with 
V(') = '(q[.']), V() =  
for q  Q, .t 6 R, and l • (R. OE)  (R, oe). It follows that 
ç+(a "t) ŒE nlin{ç(z'). 
for any a, b  Q and z t  R. However, 1' is not equal to I o 1  in general. 
We recall a definition and a result given by Mathiak in [17]. 
Definition 4.1. Let D be a skew field and (F, ) a linearly ordered set. Then a 
lnapping 1" : D*  Aut (F, ) is called an M-valuatio if the following conditions 
hold: 
r 
MV1. I = I o l,g for any a,b  D*; 



2750 H. H. BRUNGS AND N. I. DUBROVIN 

MV2. Va+b(h) >_ mill{l/(]), I/b(]))} for any a,b • D* with a + b -)/= 0 and h • C. 
If we add the syInbol oc for infinity to F and define V0(b) = oc and t(ec) = oc 
for all h • F, 0, a • D, then MV1 and MV2 will be valid for all eleInents a, b • D 
and all b • F U {et}. 
The next result follows almost directly froIn t.he previous definition; see also [16] 
nd [17]. 
Proposition 4.2. Let V • D*  Aut(F, _<) be an M-valuation for a skew field D 
and a linearly ordered set (F, <_) and let 5 be an element in F. 
Then the set Sh = {d • D I Vd(h) _> 5} is a total subring of D. Conversely, any 
total subring S in a skew fleld D can be obtained in this way for F = {aS I a • D* }, 
aS >_ bS if and only if aS c_ bS and I)(aS) = daS. The ring S coincides witb Sh 
forh=S • F. [] 
The space K{G} introduced above is also a left G-module if we define for g • G 
and 3' = '. xtlki • Iç{G} that 
g3` = :C t' (Ulkl) n t- xt; (U22) n t- xt (u3k3) -t- . • • 
where g. x t = xt',i for ui • U ç K, t' i • ll. It follows from Lemma 2.6(d) that 
t' < t < t < ... is also well ordered and hence g7 • K{C}. The group ring TC 
can therefore be considered as a subring of Q. 
If A is any subring of Q. then we define 79[0. ,4] = A and 79[n + 1, A] as the 
subring of Q generated by 79[n, AI and ail inverses of elements of 79[n, ,4] in Q. The 
union 

U 79[n, A] = 79[A1 

n=0 
is called the rational closure of A in Q. Let I13 = 79[TG] be the rational closure of 
the group ring TG in Q. 
The following result can be round in [11] (see [10] also): 
Theorem 4.3. a) The rational closure I of TC in Q is a skew field. 
b) The mapping 1  restricted fo I* is an M-valuation of I* to Aut (R, _<). 
c) The rang S = {d • I13 I Vd(x°)  x°} is an ex«eptional tank one chain order 
in  of type ( C ) associated with the exceptional cone P in the group G.  
In order to construct skew fields that contain rank one exceptional chain orders 
of type (Ck) we consider the rational closure Dk = [THk] of the group ring TH 
for the group H = gr {u,,w2,x "} (see Theorem 3.8) in Q = End K{G}3. 
Since Dk ç  = [TG] C Q and  is a skew field by the above theorem, it 
follows that D is also a skew field and S = S  D is a total subring of D. 
It follows from Corollary 1.10 and Theorem 3.8 that S is an exceptional rank 
one chain domain of type (Ck) if the following theorem is proved: 
Theorem 4.4. The total subring Sa = S  Dr is associated with the cone Pk = 
P H. 
Before this theorem can be proved, we need the result in the following lemma. 
Lemma 4.5. Let   K{}. Then 
(,)  suppd[? l ç  I(supp?). 
dD gH 



CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2751 

Pro@ Let I: be the right side in (,). Then iii order to prove (*) it is sufficient to 
prove 
(**) supp d[T] C_ }: 
for any 7  K{G} and any d  Dk = U [n, THk]. X will prove this in rive st.eps 
using induction on n for  the smallest index with d  [n, THk]. 
STEP 1. Assmne that d = xtu ¢ Hk, u ¢ U and that 
i<A 
0  ki  K for all ordinals i < A. 
Then d = xt:(uiki) for .rtu.r t" = xtœe,i  , ai ¢ U. Hence, suppd[] = 
i<A 
suppd= {x t{ l i< A} = {lh(xt') [ i < A} Ç {I(xt') l g¢ H.,i 
STEP . The inclusion (**) follows ilnlnediatelv for d ¢ T. 
STEP 3. Assulne that o, b  Dk with suppa[] U suppb[] ç I for any ¢ Iç{G}. 
Then supp (a+b)[] ç supp a[]U suppb[] ç }ç. Further, I(}') = } for g ¢ Hk 
and hence 
"PP( )[q ç ;[ = U ç("pp[q) ç U ç)= ,. 
gEHk gEHk 
STEP . It follows froln Stcps 1-3 that the statement (**) is true for ail d 
STEP 5. Assume that (**) is true for elements d  [, - 1, THk] for some n  1 
and all 7  K{G}. 
Let d = p-  [,,. THk] with p  [, - 1, THé.]. XX consider fl = d[7 ] and 
decompose fl into the sure fl = flo +fl with supp (flo) 
Th, v = p[Z] = p[£] + 
By the induction hypothesis, it follows that 
gHk gHk 
Hence, supp (pitié]) = supp (7-P[o]) ç supp TU supp (pi&)) ç ]. On the other 
hand, supp (pitié]) ç ]' since p e [n - l, THk]. If we assume that there exists 
 element h in supp (pitié]). then, on thc one hand, 
forsome gHk andsome h' supp(7) 

h = Ih' 
and on the other hand. 
 = V« (h") 

for solne g' E Ha. and solne h" E supp (fll)- 
This ilnplies h" = I@,}«9(h' )  Yv ç supp (/31) = 0, a contradiction that shows 
that supp (fl) is empty and supp (ff) = supp (flo) 
The ring [n, TH»] is generated by [n - 1,THk] and all elements p- for 
p  [n - 1, THk]{O}, and it now follows bv an application of Step 3 that (**) 
is true for ail elements in [n, THk] which completes the induction and proves the 
lemma (see also: [111). 
We now return to the proof of Theorem 4.4. 
Let d be a nonzero elenmnt in D. Since Dk ç D and S is associated with the 
cone , the element d can be decomposed as follows: 
d = xtm = qx t' with m,q  U(S), xt = xt' 



2752 H. H. BRUNGS AND N. I. DUBROVIN 

(see Definition 1.2). It, follows from (*) iii Lelmlm 4.5 with 3' = x° that 
suppd[x °]C U I('r°)= U v(g}. 
gEHk gEH 
Hence, ,(d[x°]) = l(.r °) = I( o l(x °) = l:,(.r °) = ,c t since m  U(S), and 
hence .r t = v(g), g = ,rv, «  U for SOnle elelnent g  H. 
It follows that d = (.rtu)(u.-m) for xtu  H and u-m = (xtu)-d  Dk. 
Further, u-m  U- U(S)  Dt. = U(St.), since U Ç U(S) and S  Dt- = S. 
Applying the salue argulnent.s to the elenlent d - = x-t'q -1, we conclude that 
there exist.s an elelnellt f  Hk with 9  = x -t' w for SOlne w  U. Hence, we obtain 
a decolnposit Joli 
d-t t' lq-1) -lq-1 
= (x- u,)(u,- for « = (g,)-ld- e Dt. a lz(ç) = U(S). 
This proves the first hall of condition (ii) iii Definition 1.2. if we write d = 
(qu,)(u,-.rt'), q«,  U(Sk), w-ix t' = (9') -i  H. It renlains to prove the equality 
Px,P = pu-.r'p.. 
Let w-ix t' = ,t".t for sonw u   U and t  N. Since q" is associated with , 

it folh)ws that 

Therefore, 

IP.rt'lp = Ipxt"Ip. 

P.rtuP = Pt.xtu(Y c H) = Pt.xtY t Hk 
= lPxtlp çl Hk = IPxt'IP ç Hk = II%t"IP ç Hk 

[] 
Corollary 4.6. Tbe cbain domain Sk is exceptional of tank one and of type (Ck). 

1:1 EFERENCES 

1. G. M. Bergman. Right-orderable groups that are not locally indicable. Pacific J. Math. 147 
(1991), 243-248. MIR 92e:20030 
2. H. H. Brungs, H. Marubayashi, and È. Osmanagic. A classification of prime segments in 
simple Artinian rings. Proc. Amer. IvIath. Soc. 128 (2000), 3167-3175. M1R 2001b:16055 
3. H. H. Brungs and M. Schr6der. Prime segments of skew fields, Çanad. J. Math. 47 (1995), 
1148-1167. MR 97c:16021 
4. H. H. Brungs and G. T6rner. Chain rings and prime ideals. Arch. Math. 27 (1976), 253-260. 
M1R 54:7537 
5. H. H. Brungs and G. T6rner. Extensions of chain rings. Math. Zeit. 185 (1984), 93-104. MI 
85d:16012 
6. H. H. Brungs and G. T6rner. Ideal theory of right cones and associated rings. J. Algebra 210 
(1998). 145-164. MR 99k:20113 
7. P. M. Cohn, Skew Fields, Çambridge University Press, 1995. M1R 97d:12003 
8. N. I. Dubrovin. Chain domains. Moscow Univ. Math. Bull. Ser. 1. 37(1980). 51-54. 
81g:16004 
9. N. l. Dubrovin. An example of a chain prime ring with nilpotent elements. Mat. Sbornik 48 
(1984), 437-444. M1R 84f:16012 
10. N. I. Dubrovin. The rational closure of group rzngs of left-orderable groups. Mat. Sbornik 
184(7) (1993), 3-48. MR 94g:16035 



CLASSIFICATION OF RANK ONE CHAIN DOMAINS 2753 

11. N. I. Dubrovin. The rational closure of group rings of left-ordered groups. SM-DU-254, Duis- 
burg, 1994, 96 pp. 
12. T. V. Dubrovin and N. I. Dubrovin. Cones zn groups. Sbornik Mathematics 187(7) 1996, 
1005-1019. hIR 98c:20082 
13. L. Fuchs. Teilweise geordnete algebrazsche Strukturen. Vandenhoeck und Ruprecht, G6ttingen, 
1966. hIR 34:4386 
14. V. K. Goel and S. Iç. Jain. n-injective modules and rings whose cychcs are r-injective. Comm. 
Algebra 6 (1978), 59-73. hIR 58:11016 
15. bi. I. Kargapoiov and Ju. I. Merzljakov. Fundamentals of the Theory of Groups. Springer- 
\rlag, 1979. MR 80k:20002 
16. N. lklathiak. Zut Bewertungstheorie nicht kommutativer K6rper, J. Algebra 73, (1981), No. 
2, 56-600. MR 83c:12026 
17. K. Mathiak. Valuations of skew fields and projective Hjelmslev spaces. Lecture Notes in hlath. 
1175, Springer-Verlag, 1986. MR 87g:16002 
18. B. L. Osofsky. Noncommutative rings whose cyclic modules bave cyclic inective hulls. Pacific 
J. Math. 25 (1968), 331-340. lkIR 38:186 
19. E. C. Posner. Left valuatzon rings and simple radical rings. Trans. Amer. Math. Soc. 107 
(1963), 458-465. hIR 27:3665 

DEPARTMENT OF IIATHEMATICAL AND STATISTICAL SCIENCES, UNIVERSIT , OF ALBERTA, ED- 
MONTON T6G 2G1. CaNaDa 
E-mail address: hbrungs@math.uaXberta, ca 

DEPARTMENT OF IIATHEMATICS, VLADIMIR STATE UNIVERSITh. 
VLADIMIR, RUSSIA 
E-mail address : ndubrovin@mail.ru 

GoR STR. 87. 600026 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2755-2769 
S 0002-9947(03)03270-7 
Article e|ectronically published on February 25, 2003 

ON THE SPECTRAL SEQUENCE CONSTRUCTORS 
OF GUICHARDET AND STEFAN 

DONALD W. BARNES 

ABSTRACT. The concept of a spectral sequence constructor is generalised to 
Hopf Galois extensions. The spectral sequence constructions that are given 
by Guichardet for crossed product algebras are also generalised and shown to 
provide examples. It is shown that ail spectral sequence constructors for Hopf 
Galois extensions construct the saine spectral sequence. 

1. INTRODUCTION 
A. Guichardet [3] has given two constructiols for a spectral sequence with/ïT 'q : 
HP(G, Hq(], X)) and target the Hochschikl cohonlology H°(G x« B, X) of the 
crossed product algebra A = (7 x B. where (7 is a group, B is an algebra, c is a 
represeutation of (7 by automorphisns of B and the nmltiplication in A is given bv 
(g,b)(g',b') = (gg',a-,l(b)b'). 
Here, X is a left and right A-bimodule or, equivalently, a left module over A « = 
A ç-) A °p where A °p is the opposed algebra of A. These constructions are mmlogous 
to the Hochschild-Serre constructions for the spectral sequence of a group extension. 
He has asked if the methods of Barnes [1] can be used to show that they construct 
the smne spectral sequence. 
D. Stefal [5] has given a spectral sequence, based on the Grothendiek composite 
functor spectral sequence, for the cohomology of a Hopf Galois extension. 
The three contexts have some features in common. All have a "large" algebra A 
and the category .A of A-(bi)modules, a subalgebra B, a "small'" algebra (7 which 
plavs the role of a quotient, of .4 by B and the cat.egory C of Cnlodules and a 
category/7 in which the filtered cochain complexes are constructed. Ail have a left 
exact flnct.or ¢ • .A --, C and a left exact functor ¢ • C --, /7, raid the spectral 
sequences have as t.arget the right derived functors of the composite 0 = ¢ o ¢. 
Throughout this paper, ¢, ¢ and 0 will denote these functors. 
In Barnes [1], A is an auglnented algebra over a commutative ring .q and C is 
the quotient AffB of A bv a normal augmented subalgebra. A, B and A/lB are all 
assumed to be projective as .q-modules. The functor ¢ is given for the left A-module 

Received by the editors April 30, 2001. 
2000 Mathematics Subject Classification. Primary 18G40, 16W30; Secondary 16E40. 
Key words and phrases. Spectral sequence, crossed product, comodule algebra, Hopf Galois 
extension. 
This work was done while the author was an Honorary Associate of the School of Mathematics 
and Statistics, University of Sydney. 

()2003 American Mathematical Society 



2756 DONALD W'. BARNES 

X by 
eX-- XB= {.cE .\'lbx=e(b).r fo," all bE/3}. 
For a C-module }', P} = }-c and we have ppx = X A for any A-module X. 
In this context, p has a left adjoint j • C - al, and use is ruade of the counit 
7r = jp • al - al of the adjunction. Note that pj = id • C - C. \Ve shall refer te 
this as the HS context. 
In Guichardet's paper, A is the crossed product G x«/3. where/3 is an algebra 
over the field .ff, C is the group algebra JïG and the fimctor p is given for the left 
A«-module X by 
OX = X B = {.'r E X I b.r = :rb for all b E B} 
with the action of G given by g.r = (g, 1)x(g -1, 1) for g E G and x E X B. For the 
.¢G-module 
e"}" = }-G _ {y E }'IgY = Y for all g E G}. 
In this context, çb does net bave a lefl a(tj()int j. The assumption that .ff is a field 
can be weakened if in seine places we tel)lace "injective'" by "relativelv injective". 
\Ve require that . is a c(mmmtative ring and that B is )ï-projective. \Ve shall refer 
te this as the G context. 
In Stefan [5], C is a H()i)f algebra over the field ., A is a C-comodule algebra 
and B is the subalgebra of coinvariants. Thus we have an algebra morphisln XA " 
A -- A  C making 4 a right C-comodule, and 
B---Aç°c= {«,E A I,X4()---® 
Stefan refers te this situation as "'the extension ,4/13"'. It is assumed te be C-Galois. 
which we explain in section 2 below. As in the G context, for the A-bimodule X, 
we set px = X B. This is ruade into a right C-module using the action defined 
by Stefan [5, Proposition 2.3] and explained following Lemma 2.1 below. For the 
C-module Y, we put ¢;'1" = }-c. Again, we veaken the &sumption that .ff is a field. 
\Ve require that . is a commutative ring and that A. B and C are )ï-projective. We 
refer te this as the S context. It generalises the G context since the crossed product 
algebra A = G × « B becomes a C-comodule algebra if we set x.4 (9, b) = (9, b) ® 9 
for 9 E G and b E/3. If in the HS (Hochschild-Serre) context, .4 is a Hopf algebra. 
we mav regard it as a C-comodule algebra with the comodule strncture given bv the 
conmltiplication of A followed by the natmal homomorphism .4 ® 4 -- .4 ® AffB. 
A left A-module X mav be regarded as a 1)imodule by setting xa = e()x for a E _4 
and x E X, where e is the augmentation. This does net change çX = X , new 
defined as {x E X ]bx = xb for all b E B}, ner does it change the Rqcp(X), although 
it does change the injective modules used for their calculation. 

2. PRELIMINARIES 

We follow the notation for comodule algebras used in Schneider [4], with the 
exception that we denote the Hopf algebra by C, reserving the symbol H for coho- 
mology. Thus we have the comodule structure map, AA : A --, A ® C and express 
the image of an element a E A bv A.«(a) =  a0 ® al. The comultiplication 
Ac : C  C0,C is written Ac(c) = yc ®c2. The augmentation of C is denoted 
by e and the antipode by S. The canonical map ean : A ® A ---* A ® C is defined 
by can(a ®B a') =  aao ® al . That A/B is a Hopf Galois extension means that 
ean is invertible, which we always assume. Thus for c E C. there exist elements 



SPECTIAL SEQUENCE CONSTIUCTOFIS 2757 

ri (c), li (c) E A, hot mfi<luely detern,ined, such that tan- 1(1 ®c) =  ri (c) ®B li (c), 
which is unique. \Ve shall need thc identities proved in Schneider [4. Ptemark 3.4(2)]. 
Throughout his paper, Schneider assumes .ù, to be a field, hut his proof of the iden- 
tities makes no use of that assumption. For the convenience of the reader, we list 
the identities here. 

Lemnla 2.1. For all a  .4. b  t3 atd c, c'  C, t]e followin 9 identifies ]old: 
(a)  bri(«) ®B li(«) =  ri(c) B li(«)b, 
(b) 
()  («)h(«) = («). 
(d) r,(«).i(«)oli(«) = ,',(c)li(c)c2. 
() 
() 
Following Stefan [. Proposition 2.], we use the above relations to d«fine a right 
Cmodule structm-e on X 
we put .r- c =  ri(c)xli(c). This is well defined since  ri(c) B l(c) is a well- 
defined elelnent of A @B A alld b.r = .rb for ail b  B. Froln 2.1(a), it follows that 
x. c  X B. If a left açtion of Ç on X B iS preferred, o11(' lllaV be defined by setting 
c- = x-(Se). The sertion of LOlnlna 2.2 below holds for this lef action provided 
that the antipode S is bijective. 
Lemma 2.2. For an Aç-modle X . ( X e) c = X A. 
Proof. For z  X A and c  C, we bave 
by 2.1(c). Thus z  (XB) c. Conversely. if x  (XB) c, then by 2.1(b), 
for all 
For the crossed product algebra A = G x B, the canonical lnap is given bv 
b 
Cl((,)('.v'))=(, )(,') =('.@()'). 
In particular, can((9 -1, 1)B (. 1)) = (1.1): so we can take r(9) = (-. 1) and 
l(9) = (9.1)- The right a.«tion of C G on X B be«onmsz-9 = (9-1.1)z(9.1) 
for z  X and 9  G. Converting this to a left action gives  • z = (, 1)z(9 -1. 1). 
which is the action used in the Guichardet paper [3]. 
Note that, in the G context,..4 « = A  A °p is free as a right B«-module. In 
the S context, we assunle that A is fiat as left and right B-module. If tlmn follows 
that A « is fiat  right B«-lnodule. In the HS context, we assume that A is at least 
projective as right B-module. At some points in [1], the stronger assunlption that 
the modle quotient A/B is projective as right B-module is use& In all the contexts, 
by Barnes [1. Lelnlna 1.4.3], everv il0ective left A- or 4C-module is injective as B- 
or B«-module. Further, every il0ective of M or C is injective as .q-module. 
For any .q-module X, the A-module coinduced froln X is the module X* = 
Hom.(A, X) wit.h the action (af)(a') = f(a'a) for f  X* and a,a'  A. (For the 
coinduced right lnodule, the action is given by (fa)(a') = f(aa').) If X is itself a 
left A-module, then the nmp « • X  X* defined by («.r)(a) = ax for a" q X and 



2758 DONALD W. BARNES 

a E A is a .R-si»lit A-module monomorphism. X* is a relatively injective left A- 
module. (See for example, Barnes [1, Lenuna II.2.4, p. 20].) A module is relatively 
injective if and only if it is a direct sunmmnd of a coinduced module. By Barnes 
[1, Lenmm. II.3.9, p, 28]. if A is right B-projective, then everv relatively injective 
left A-module is relativelv injective  left B-module. Thus in the G context, every 
relatively injective A«-module is relatively injective as B«-module. In the S context, 
to get this conclusion, we nmst strengthen Stefan's assumption that A is left and 
right B-fiat t.o A being left and right B-projective, although as t.he next lemma 
shows, this strengthening is mmecessary if, as in [5], if is sumed that  is a field, 
since then, every module is -injective. 
Lemma 2.3. If X is -injeclive, then the coinduced module X* is injecive. 
Proo: Let i • I"  IV be a monomorphism and a • I"  X* a honomorphism of 
A-modules. 
i 
X* 
We want to construct a. homomorphism fl • II"  X* such that i = a. For 
v  I , we bave (tv  Hom.(A, X); so for a  A. we lmve (v)(a)  X.  m- 
regard a as afimction A x 1"  X, writing a(a,v) for (av)(a). For b  A, we 
have a(bv) = b(av). So (a(bv))(a) = (av)(b); that is, (ab, v) = a(a, bv), and in 
particular (a. v) = a(1, av). Putting (v) = (at,)(1) creates the diagram 
of -modules. Since X is -injective» there exists a -homomorphism  • I1  X 
such that Bi = . Define " II"  X* by (w)(a) = B(aw). Then for b e A. we 
have 
/(b)() = (b) = (,,,)(b) = ((,,,))(); 
so/3 is an A-module homomorphisn.  have 
(,,)() = $((i,,)) = ((,)) = (,,) = (,)() 
and fil = a. 
We have defined a right C-module tion on X u for any leh A«-module X. 
need another description of that action in the case where X is a coinduced module. 
Lemma 2.4. Let V be a R-module and let I'* = Hom(A , V) be the coinduced A - 
module. Then V *B is isomorphic to the coinduced C-module Hom(C. Hom(A, 1")). 
Proof. The tion of a @ a'  A « on f  V* is given by 
(( e ')I)( e )= I(( e )( e '))= 
for x, y  A. Now 
Y*= {f  Y* I (be )I = ( e b)I for fi   } 
= {f  V* f(xb@y) = f(a" @ bç) for ail b B and x,y in A}. 



SPECTRAL SEQUENCE CONSTRUCTORS 2759 

Thus 1"* B can be identified with Hom(A®u A, V), and so, using the canoifical map, 
with Hoin(A ¢_ C, 1;) = Hom(C, Hom(A, V)). We calculate the right C-modnle 
action Oll Hom(C, Hom(A, V)) induced by these identifications. For f  V *B and 
c G C, froln the A%a.çtion on 1", we get fc = 7,(ri(c) C' li(c))f. Thus 
(fc)(x ® g) =  f(xr,(c) ® li(c)g). 
For f • Hom(C, Hom(A, V)) we have the corresponding f' • Hom(A ® C, 1") 
given by f'(o ®c)= f(c)(a) and f" • Hom(A ®8 A. V) givell by 
f"(x ®B U)= f'(ean(x ®B y))----  f'(xgo ® U)- 

Thus, 

f(c)(a) = f'(o®c)= Zf"(ori(c)®Bli(c)). 
So for c' • C, fd is given bv 
(fc')(«)(a) = Z(f"«')(ari(«) ®B li(c)) 
=  f"(""(«)"(ç') ®- 
= f"(ari(c'c)@.li(c'c)) by 2.1(f) 
= f(d«)(a). 

Thus the action (fc')(c) = f(c'c) is that of the coinduced right C-module. [] 

The next result strengthens Stefan [5. Proposition 3.2]. The corresponding result 
in the HS context follows easily froln the fact that every AffB-module is an .4- 
module and that QB is a submodule of Q. 

Lemma 2.5 . Let Q be a relatively injective A¢-module. Then QB is a relatively 
injective C-module. If Q is injective, the QB is injective. 

Pro@ Q is a direct smnmand of some coinduced module I "* = Hom.(A , 1"). So to 
prove Q8 relatively injective, it is sufficient to show that Hom.(A «, V) B is relatively 
injective. But Hom(C. Hom(A. 1")) is relatively injective; so by Lemma 2.4, V *B is 
relatively injective. Thus QB is relatively injective. If Q is injective, we nav take 
V = Q. Since Q is .¢binjective, by Lemnm 2.3, Holn(A. Q) is also .R-injective and 
Hom(C, Hom(A, Q)) is an injective C-module. Thus QB is injective. [] 

3. SPECTRAL SEQUENCE CONSTRUCTORS GENERALISED 

In Barnes [1, Chapter III] in the HS context, a spectral sequence constructor for 
(¢, ) was defined to be a functor F from ,4 to filtered cochain COlnplexes in T) such 
that 

(1) F is exact (in every filtration). 
(2) F is acyclic on injectives; that is, if Q is injective, then H'(FQ) = 0 for 
n > 0 and for all p. Hq(°FPQ/°Fp+IQ) = 0 for q > 0. 
(3) E°F is exact on C. 
(4) The inclusion i" X B --, X induces isomorphislns Eï°F(X B) --, Eï°F(X) 
for all p and ail X • .4. 
(5) H°F is naturally isomorphic to 



2760 DONALD W. BARNES 

Here, using the fact that A//B-nlodules are A-modules, that is, using the adjoint 
j to b, we c'onstruc't a flmc'tor F : '" 
1  fi'Olll Cto cochain complexes in . This 
cannot be done in the G or S contexts. So we nmst include the cochain complex 
flmctor as part of the structure in our definition of a constructor. If F is a filtered 
cochain complex, we denote the colnponent of total degree n bv F, the pth filtration 
bv F', the submodule of filtration degree p and C'olnptementary degree q by F pq 
and use similar notation fbr the t.erlns of its spectral sequence E(F). e always 
assmne that "F ° : "F and that 'F +1 : 0. The following definition generalises 
the Olle quoted above. 
Definition 3.1. A spectral seqllcnce constructor for the pair (, ) is a quadruple 
(F. F. /. 7), wherc F is a functor from A to filtered cochain complexes in , F is a 
fimctor ffoto C to cochail c'omplexes iii , I I is a natural isonlorphisln l ("' F)  F& 
and 7 is a natural isomorphism H°(F)  ç such tha.t 
(1) F is exact (in everv filtration). 
(2) F is acyclic on injcctivcs: that is, if Q is injeçtive, then H"(FQ) = 0 for 
n > () a.nd tr ail p, Hq('FPQ/"FV+Q) = () for q > 0. 
(3) I" is exact an(t a.«y«lic on inje«(ives. 
From (1), il follows that "FV'F + is exact for all p, r. From 7 being a natural 
isomorphism and (3), it follows that we have a torique fanfilv of natural isomor- 
phisms 7 p• Ht'P + RPç, the right d('rived flm«tors of , «ommuting with colnmct- 
ing homomorphisms, and with 70 = 7- k, denote this family bv 7. Furthermore, 
H'F = E°°F._ = E°F = H°Eï)F = H°q-iF, 

and SO 

"H°(71) H°F = 7 = O: 
that is, "7H°(7/) is a lmtural isonorphism flOlll H°F to 0. It follows that H  F = RO 
for all n. 
X now prove thc results corresponding to Barnes [1. Theorem III.2.3. p. 42] 
and the lelnm leading up to that theorem, begimfing with the analogue of [1. 
Theorem III.l.5, p. 38]. 
Lemma 3.2. Let (F, F,I.) be a spectral seq,ence con.tructof There exists a 
unique famil of natw,'al t','ansformations ,1 pq " Eï ( F)  F" ( H q¢) coin m uting with 
connecti9 homomorphi.s'ms and with I1 pO = I1 p. Ile ]Pq are atural isomorphisms. 
.sati.sg I]P+l"qdï q : (-1)qgl] pq. where d is the differential 4 F. and induce 
isomophisms 
(,'")- () 

and 

HP(q°q) - Eq(F) --> (R'b)(Rq¢). 

Pro@ F is exact by assumption. So {F"(Rq¢)[q = 0.1 .... } is a connected sequence 
of fun«tors, as is {Eï(F) = Hq(F"/F"+)]q = 0.1 .... }. Both vanish for q > 0 
on injectives. By dimension-shifting, it foltows that there exists a torique family 
of natural transformations 1 "q - Eï(F)  ['P(/ïqçD) extending the given tra.nsfor- 
mation qP • Eï°(F) ---, FP¢. Since /p° is a natural isomorphism, all the t' are 
natural isomorphisms. The argument of [1, pp. 38. 39] applies unchanged to give 
the result. [] 



SPECTRAL SEQUENCE CONSTRUCTORS 2761 

Definition 3.3. A natm'al transibrmation ( • (F. F. q, 7) -- (F'. F'. q'7') of con- 
structors is a pair F " F -- F  and çFÇ -- I , Of natural transformations such that 

the diagralns 

COlllnlllt (L 

VVe shall onfit thc subscripts from çF and çr- Out next lemma is easier than 
[1, II.2.2] in that the transformation ç • F -- F' is given instead of having to be 
constructed. 

Lennna 3.4. 
constructors. 

Let  • (F, F, '1, 7) -- (F', F'. 1', 7') be a natural tra,fformatton of 

Then the diagram 
:ït ( r )  
-- FP Rqc 
Eïq(F,) '/ , F'PRqdp 

commutes for all p and q. 

Pro@ çH°ç is a natural transformation of connected Se(lUellCeS of fimctors. Since 
in dimension q = 0, we have R°¢ = I//P0(I/P0) -1, tV dimension-shifting, we have 
çq¢ = ?]tpqÇ(?]pq)-i for ail q.  
Theoreln 3.5. Let ç • (F. F. q, 7)  ( F. F, q', 7 ) be a natural traufformotio, of 
constructors. Then ( induces a natural isotorphism. çE " E(F)  E(F ) of their 
spectral sequences, that is, (.q • Eq(F)  Eq(F ) is a natural isomorphism for 
all r  2 and all p. q. 
Proqf. Bv assumption,  " H°(r)  , and 7' " H°(F')   are natural isomor- 
phisms, and 'H°(ç) = . Therefore H°(ç) = (')- is a natural isomorphism. Bv 
dimension-shifting, it follows that HP() " H  H  is a natm'al isomorphisln 

for ail p. The diagram 

is, up to sign, a commutative diagram of cochain COlnplexes. (If q is odd. q and 
q' anticommute with the differentials.) Taking H p of this, ve get the commutative 

diagram 

in which '1, '1' and HP(ç) are natural isomorphisms. It follows that çq is a natural 
isomorphism and so, that ,,cm is a narreal isomorphism for all r _> _.° [] 



2ï62 DONALD "vV. 13ARNES 

GI. ICHARDET S FIRST ('ONSTRUCTOR 
Foi" the Ae-lnodule X, Guichardet defines the double colnI)lex 
içpq(X ) = (2r(av+I,Hom(®V+IA«,X))B)a 
with apI)ropriately defined differential, where .T-(b, V) denotes the set of furie- 
tions rioto the set U to the set I . Following Guichardet [3], we set I'(X) = 
Hom(® '+IA , X) with differential 
! 
d,f(oo ® o'o ..... o,+, ) (/ri+l) 
! 
= a0,f(o ®(4 .... ,o+  
( ' , , 
+ -- 1)i+,f(ao ® o' o ..... aiOi+l @ ai+lai,., • an+l @ an+ 1 
0 
which gives a relatively injective resolmion l'(X) of X in A. Also following 
GuMmrdet, we put Pn = ®'+I-R(,' with action g(9o ®... ® .On) = .O.Oo ® ... ® .O.On 
and set d(go ®... {" g,,) ' ' = 
= Yi=0(-1) (9o®...'i. -®g,,) and e(90) 1. This makes 
Po a ff'ce resolution of . iii C. We then have 
lçoq(X) = tlom.G(Po, Iq(X)B). 
Expressed iii this way, it is the Grothcndiek repeated (relatively) injective resolution 
construction foi the spectral sequcnce of a composite flmctor discussed in Barnes 
[1, Chapter VIII, with Hom.(Po, ) used as the relatively injective resolution fimctor 
on C. For any relatively injective resolution flmctor I" and any projective resolution 
P.. setting KPq(X) = Hom.c;(Pp, Iq(X)B) gives a constructor (K.F, 0,7) with 
F = Hom.c;(Pv, ), r = id and 7 = id. The spectral sequence constructed is 
independent (from the E2-1evel onward) of the choice of I ° and of P.. 
This constru«tor mav also be regarded as a.n adaptation of the Cartan and Eilen- 
berg pair of resolutions constructor dis«ussed in Barnes [1, Chapter VI]. Since 6"7- 
modules are hot A«-modules, we cmmot use Homa, (Pv, Iq) as in the HS context. 
but use instead Homc(Pp, (Iq) B) which, iii the HS context, is essentially the smne, 
Stefan in [5] establishes the conditions foi" the Grothendiek composite flmctor 
spectral sequence. To obtain a. spectral sequence constructor, we bave merelv to 
assign ffmctorially the resolutions used iii the construction. If we assume that A is 
left and right B-projective or if we assume that Jï is a field, then we can use the I n 
defined as above and an,v right C-module projective lesolution P. of 

5. GUICHARDET'S SECOND CONSTRUCTOR 
For his second construction. Guichardet defines a filtration on the normalised 
standard COlnplex °N(A. X) where N(A, X) is the subspace of Hom.n(®A. X) of 
functions f for which f(a ..... a) = 0 if anv of the ai is in .1, and 
df(al,. .. ,an+l) 
Yt 
= al,f(a2 ..... an+l) n u -(--1)i,f(al ..... aiai+l ..... an+l) 
i=1 
+ (- 1)'+1 f(al ..... (-In)On+ 1 . 



SPECTRAL SEQUENCE CONSTIRUCTORS 2763 

The filtration on this colnplex is given by defining n_N° 
tobe the subset of those fllnctions f satisfying 
f(al .... , aq, glbt,..., gpb v) 
--i (bl)O, L1...gp(b2 ) o,;pi(bp_i)bp. 
= f(o1 ..... aq, gl,''', gp)OZg2g3...g p "'" 
Gnichardet t.akes for F the normalised standard complex and constructs a natnral 
transformation from E°(N) t.o ['b which, in [3, Lennne 3.11], he shows is a nat- 
ural isomorphism. The conditions for a spectral sequence constructor are clearly 
satisfied. The purpose of this section is to gcneralise this to the S context. 
To nse the nornlalised standard COlnplex in the S context, we must ilnpose a 
further condition on the algebra A. The theory of the norlnalised standard coin- 
plex (Cartan and Eilenberg [2. p. 176]) reqnires that the quotient A = 4/.ql be 
projective as .-module. We assnlne this in this section. Equivalently, we assume 
that there exists a .-linear map • " A --, .q snch that •(ki) = k. This condition 
alwa.vs holds if . is a field or if, as in the HS context. A is ail anglnented algebra. 
An equivalent definition of nNP, also given by Guichardet, is memfingful iii the 
S context. So we use it hele but with sides revel'sed because of onr nse of right 
COlnodnle algebras and right C-nlodnles. Fol" p >_ 1, we define A "p to be the subset 
of 'N(A, X) of those filnetiolls satisfying 
(5.1) f(bal,. ..,a_i,an) = bf(al,....a) 
and 
(5.2) f(al,...,ai-ib, ai,...,an) =f(al,....ai-l,bOi,...,an) lori=2 ..... p 
for all a ..... a  .4 and b  B. 
The normaliscd standard complex °N(C, )') for a right Cmodule t" is that ob- 
tained by treating )" as a bimodule with left action c- g = •(c)9. Thus, N(C is the subspace of Hom.(®*C )') of functions f for which f(c ..... c) = 0 if any 
of the ci is in .fil. with the differential 
n 
df(cl,.. . ,c+) = e(c)f(c2 ..... c+_)+ Z(-1)if(«l,...,«iCi+l,... ,c,+) 
i=1 
+ (--l)n+lf(cl ..... Ch)Ch+l. 
We put T pq = PN(C, q]v(13, X)) and write N pq for P+qN p. Note that, although 
N(B. X) is hot, in general, a Cmodule. this does define .-lnodules T pq. 
For f  Nvq(A, X), we put 
qPq(f)(Cl ..... Cp)(bl,..., bq) 
= Zri(cP)...ri(c)f(li(c),...,l,(cp),b,...,bq). 
That O2Pq • N pq -- T pq is a well-defined .-linear map follows from the next lelnlna. 
We shorten the notation by writing (ï for a string al,. • •, Oq of elements of A of any 
length. Ve flirther shorten notation by onfitting unnecesary subscripts from the 
(c),h(c). 
Lemma 5.3. If   NPq(A. X), then for j = 1 .... ,p. and ail   A. b  t3 and 
C, C1,...,C p  C 
(1)  r(cj) . . . r(cl)f(l(Cl), .... l(cj),) is independent of the choice of the r(cj) 
and l(c). 



2764 DONALD ç\. BARNES 

(2) '. b'r(cj)'r(cj_l)...'r(cl)f(l(Cl) ..... 
=  r(cj)...r(cl)f(l(c) ..... l(cj)b. 6). 
(3) y r(c)'r(c)...r(c)f(l(ci) ..... l(cy)l(c).) 
-  r(cjc)...r(cl)f(l(c) ..... l(cjc).ç). 
Proof. For o. o'  ,4. we put 
gj(a @ a') =  a,'(cj_)...r(q)f(l(c) ..... l(cj_),a',g). 
Bv the condition (5.1), g is well defined. Thus (1) holds forj = 1. Also. by putting 
, -}B a'=  br(Cl) eB /(Cl) =  r(Cl) @B 
bv Lelmua 2.1(a). we see that (2) holds folj = 1.  use induction over j. 
For I <jp, wehave 
=  or(cj_)...r(Cl)f(l(c,) ..... l(c_),bo'.6) 
1)v the induction hyl)othesis that (2) holds for j - 1 and condition (5.2). Thus 
gj is well defined. Putting « '}B o' = r(cj)$ l(c) gives the assertion (1). 
Putting a OEB a' = b,'(c)@B I(cj) and using Lemlna 2.1(a) gives (2). Putting 
a 'B a'=  r(c)r(cj)@B l(cj)l(c) and using Lelmna 2.1(0 gives (3). 
Lemma 5.4.  defines a nalural cochain map o'" " E'"  T v'. 
Pw@ If f  nNp+I, then 
¢çq(f)(c,....cp)(b ..... bq) =  r(cp) ...r(cl)f(l(Cl) ..... l(%)b, 1,b ..... bq) 
= O. 
Since "ç0 = NP/"N p+, Pq defines a .q-linea.r lnap ¢q • Eg q  T pq. Consider 
the eXplessiol for (daf)(l(q) .... , l(cp), b ..... bq+). For those terlns in which 
the string of/(ci)'s is reduced in length, we get bi in the pth place: so those terres 
are 0. Thus. 
( P'q+ l d 4f )(c1 ..... Cp )( bl ..... bq+l) 
= (-1F ,'(%)...r(q)f(l(c),....l(%)b ..... b+) 
+(-l:+>(«)...y( .... (Vb+) .... ) 
+ (-:++ («)---/(I(«,) ..... )+ 
= (-1)PdB((Puf)(q,...,cp))(bl ..... bq+) 
by applying Lelnlna 5.3(2) to the first terln. The result follows, the naturalitv being 
obvious. 
 are trying to construct a spectral sequence coustructor using Aç4 = N(A. ) 
with the Guichardet filtration as the filtered complex functor. Clearly, we tan set 
F = N(C, ) and 7 = id • H°(C, )  .  still need a natural isolnorphisln 
q • E°(A4)  Y. Applyilg H q to the natural cochain map ¢" gives a natural 
map qPq • Eï  PN(C. Hq(B. )) = FP(Hq(B. )).  nnlst fil'St show that 
is al isomorphisln of cochain complexes. 



SPECTRAL SEQUENCE CONSTRUCTORS 2765 

Lemma 5.5. ,l "q " Eï(X) --, F°X B iS a map of cochain complexes. 
Pwof. An elelnelt of Eï°(.¥) is rel,l'eselte(l 1,y a fllll('tiOll f  PNP(A. X) such that 
f  P+INp+I. For f  PN p, every terln t in df satisfies (5.1) and (5.2) for ail i except 
the terre t(a ..... ap+) = f(al,..., ap)ap+l [or whi«h (5.2) lnay rail for i = p + 1. 
Thus the l'eqlfirPlnelit that dr  p+INp+I ilnpOses the OlW extra conditiol that 
f(al ..... apb)ap+ = f(a .... ,ap)bap+, that is, f(a ..... apb) = f(a ..... ap)b. 
For su«h an f. we have, wriling  ri)l" c 1 ..... Cp+l, 
(P+'°dAf)( =  r(cp+)...r(c)«f(l(Cl) ..... l(Cp+)) 
=  r(«+l).., r(«l)t(«)f(t(«) ..... 
+ (-)'r(«»+)... ,'(«l)f(tt«) ..... («)t(«+l) ..... 
+ 
By Lemma 2.I(c), 
= (Cl) r(%+l)...'(«)f(l(ç) ..... 
By Lcnmm 5.3(3), 
(-1)(ç+l)... («)/(t(«),..., («,)t(ç+l) ..... (ç+l)) 
= (-)',.(«+)...,.(«.«+)... ,-(ç)f(t(«) ..... (ç«+) ..... 

ï r(«p+l).., r(«)ï(l(«),.. 
= (Zr(«p)...r(«l)f(l(«l) ..... l(«p))) "Op+l. 
Thus kP+LOdAf = dcq2P°f and the result follows. 
For g  PN(C, xB), we define dpg  P/'(A..\-) by 
((I)g)(c/1 ..... c/p) = Z G10"" 0 1 1 
• 
o 
wl'it.ilig the comodule structure indices as supel'scripts, 
Lemma 5.6. Oi, g e PNP(A. X) and d_4(Oi, g) 
Pro@ For b  B, we have 
sincc AA(bal)= OE haï ® a l. Thus condition (5.1)is satisfied. Also, 
( Opg) (a ..... aih, ai+l,..., 
__ Z(lï.. 0 0 0 1 llp) 
-- a i bai+ l •.. 
• ap9(a,.... 
= (dPg) (al ..... 

[] 



2766 DONALD V . BARNES 

Thus (5.2) is satisfied for ail i and g E I)NI)(A,X). Since g(a ..... @)  X , 
(9) (al .... , avb) 
.... %b(1,...,) 
= aï.. 0 1 aç)b ...... ap)b 
• ap9(al,..., 9(al 
and it follows that dag  P+INp+I 
Lemnla 5.7. For 9  PN(Ç, xB), PO 
Pro@ For any .-linear flmction t • C  X, setting u(a  a'  c) = aa't(c) for 
a. a'  A and c  CdefilleS a .-linear function a • A @B A @ C  . Bv Lemma 
2.(a), 
 r(«)(c)°tq(«) ) = .,,( r(c)/(«)0  (c)) 
= 
=  F(cl)I(cl)t(c2) 
=  ¢(cl)t(c 2) by Lemma 2.1(c) 
= t(c). 
Using this with t(c) =  l(c2)°.., l(%)°g(c,/(c2),..., l(cp)l), we have 
(çP0)(C1,... Cp) = '(Cp)...r(Cl)()(/(Cl) .... 
= r(Cp)...r(Cl)I(Cl)O...l(ep)°(l(Cl) 1 .... ,l(cp) 1) 
=  r(%)...r(c2)l(c2) ° . ..l(%)°g(c,l(c2)  ..... /(%)1). 
Repeating this argument gives the result. 
Lenlma 5.8. If f  PNV(A,A ") and dal  P+I'p+I(A,ç). then po f = f. 
Pro4 Setting u(a  a') =  ar(@_l)...(l)f((l) ..... (-1).') defines a 
.-linear function u" A @B A  X by Lenlma 5.3(b) and condition 5.2.  have 

1Repeating this argument 

ai)r(ai))...O 1 ,-(l)f«(l),....,(1)) 
0 0 1 l(alp)) 
ai)-lZ(apr(ai)) @B 
@-lU(1 ®B ai)) by Lenlma 2.1(b) 

E aï " " " ai)--lO ir(a ..... ) (I)(,(I),.. ,/(apl--1), ai))" 
gives t.he result. [] 
Corollary 5.9. The ri I)° are isomorphisms. 
Pro@ Eï ° is the set of f  I)N(A,X) with dal  I)+INI)+I (A, x), and i11)° is the 
restriction i)01Eï ° --' I)N(C, XB). By Lemma 5.7, it is surjective and, bv Lemma 
5.8, it is injective. [] 



SPECTRAL SEQUENCE CONSTRUCTORS 2767 

Theorenl 5.10. Suppose  = A/fil is projective as .ff-module a'nd that A/B is 
projective as lift B-module. Tben (NA,F,I, ) is a spectral sequence coustru«tor 
Pro@  have to show that the couditions (1), (2), (3) of Definition 3.1 are sat- 
isfied. Since 
"NA (X) = Holn(@(.), X) = Hom(.. H,,ln(@ - .4, X)) 
and . is .-projective, N is an exact flmctor. A flmctiou f ¢ n satisfies the 
further couditiou f(ba .... ,a) = bf(a,...,a) for all b ¢ B. In particular, 
f(a,...,a.) = 0 ifa ¢ B. Thus 
nNÀ (X) = Holn(n A, HomB(A/, X)). 
Thus nN is an exact flmctor. Similarly, 
(X) = nom(.4, no,,,(A/, X)) 
and by iuduction over p, N is exact. Thus condition (1) holds. 
Let Q be an i0ective A«-mo(hfle. Then H([4Q) = 0 for n > 0 by the usual 
theory of the nornmlised standard complex. X have to show that 
u ¢',v O'-x[ +  O) = 0 

forq>0. But 

Hç(°NPAQ/°N+Q ) = EïA'A(Q)  PN(C, Hç(B,Q)) 
by Corollary 5.9. But Hç(B, Q) = 0 for q > 0 since Q is injective as B««nodule. 
Thus conditiou (2) holds. 
Sin«e F(}')= N(C, }'), «oudition (3) holds. [] 
111 the discussion of the filtered nornlalised complex in Barnes [1. Chapter IV], 
the corresponding extra assumption that A/B be projective as right B-module was 
needed. If in the HS context, .4 is a Hopf algebra, then it cau be regarded as a 
A//B-COnlodule algebra. The Guichardet filtration is not the saine as that given by 
Hochschild and Serre, but by the result of the next section, the two filtratiollS give 
the sanie spectral sequence. 

t3. UNIQUENESS OF THE SPECTRAL SEQUENCE 
As in BarlleS [1, Chapter X], we construct for each cardiual a, a coiffe fmlctor 
which, restricted to the subcategory .A of objects of cardinalitv less than a, is a 
spectral sequence constructor. (This use of the subcategory .A is necessary because 
a coiffe functor with injective lnodel /I aud injective basis (M, U) only has the 
desired properties with respect to lnodules embeddable in M.) From the existence 
of this cofree flmctor, we deduce as in [1, Chapter X], that all spectral sequence 
constructors construct the saine spectral sequence. We need one technical lemlna 
to get around the diflïculty caused by C-modules not being A%lnodules. For this. 
we again need the assumption that ri is q-projective, that is, that there exists a 
.ff-module homomorphism e : A -- q with e(1) = 1. 
Lemma 6.1. For every C-module Y, tbere exists an ijective A«-module Q such 
that Y can be embedded in Q. 



2768 DONALD ,V. BAF{NES 

Pro@ We first lnake Z = Hom(A, Y) a C-module l»y defining (f. c)(a) = f(a)c for 
c  C. a  A and f  Hom(A. Y). X construct an embedding i : }" + Z bv setting 
(ig)(a) = e(a)g for g  Y and a  A. S« defined, i is a C-module homomorphism, 
because 
(i(c))(«,) = ()9« = («(«,))c = (@(,,)« = ((+) • «)() 
tbr g E G, a E A and f E Hom(A. }'). It is clearly injective. 
Next, we use the standard embedding of Z in the coimhlced C-module Il = 
Hom.(C, Z), defining « : Z + II l»v setting a(z)(c) = zc for z E Z and c 
now ha.ve an embedding of Y in Ho»re(C, Hom(A. }')). By Lemma 2.4. we bave an 
embedding of }" in X B where X is the coinduced A:-module Hom(A «, }'). Taking 
any embedding of X in an injcctive A'-module Q, we get an embedding of 
 use the thcory of cofree filnctors develol»ed in [1. Chapter X]. Out spectral 
scquence constructors consist of two ftmctors and txvo natural transformations in- 
stea<l of the single flmctor used in the HS context. To accommodate this. we shall 
sav that the pair (F, F) of fimctors. F defined on A and F defined on C. is simple 

«ofrec on the 1)asis (5I. r, V) if F is cofree on (M.U) and V is cofree on ((M). V). 
Theorem 6.2. Let C be ( Hop.f «l.q«bra over .q and let A be a rigbt C-comodule 

algebra with B = 4 `c. Suppose ,4. B and C are .-projective atd that A/B is C- 
Galois. Let ,,4 be the categorg of left A"-modules, C tbe category of right C-modules 
atd let 73 be the categorg of .-moddes. Let  : ,A ---+ C atd ¢ : C -+ 73 be the 
flmctors defined bg ¢(X) = X B and '(}') = I "c for X  ,,4 atd Y  C. Suppose 
.4/.ql is .q-projective ad tbat .4 is both lefl attd right B-fiat. Then. for atg cardinal 
n. there exists a simple cqfree pair (T. F) with ijective model M ad injective basis 
(M. U, I'). and atural transformatiots 1.7 such that on the subcategory +-l of 
objects of gt of cardimlit.q less tha . (T. F, q,"f) is a spectral sequetce cotstructor 
for (¢, ). 

P.v@ By replacing  bv a suitable larger limit cardinal, we nlay suppose that everv 
object in A¢, bas an injective resohltion in A, and that every object of C likewise 
has an injective resohltion in C. There exists an injective module X in A such 
that every object of A can he embedded in X. Likewise, there exists an injective 
module I" in C StlC}l that e»vry object of C can be embedded in Y. Bv Lemma 
6.1. there exists an injective lnodule Q in M such that }" can be embedded in (). 
Putting Al = X @ Q. we obtain an injective module M such that every module 
in A has an injective resolution, all of whose terres can be embedded in M, and 
every module in C+ h an injective resolution ail of whose terres can be elnbedded 
in (M). By [1. Lemma X.3.2. p. 101], there exists a silnple cofree functor F from C 
to cochain complexes in  wih basis ((M), V) for some injective I', and natural 
transfornmtion  : H'(F) + R' which, on Ca, is a natural isomorphimn. The 
construction of T and the proof of the result now follows exactlv as for [1. Theorem 
X.5.3, p. 107].  

Theoreln 6.3. Let C be a Hopf algeb'ra over . a'td let A be a rtgkt C-comodule 
algebra with B = A c. Suppose A. B ad C are .-projective and tbat A/B is 
C-Galois. Let A be tbe category of left A«-modules. C the categorg of right C- 
modules a'd let 73 be tte category of .ff-modules. Let c : .,4 -- C otd l, : C -- 73 
be tbe flmctors defited bg d)(X) = X  atd /,(}-) = }-c .for X  A and }  C. 



SPECTRAL SEQUENCE CONSTRUCTORS 2769 

Suppose A/.¢I is .-pvojective and that A is botb left and right B-fiat. Suppose 
I 
F = (F, F-, I', 7') and F' = (F', F,, I-, 7-) are spectral .seque'nce co'n.structors 
for (¢, ). Tttea F and F  cottstruct caoztically isomorpbic spectral sequences fTom 
the E2-1evel o,ward. 

Pro@ The mgument of [1, Thcorem X.5.4, p. 109] al»plies unchmged. 

EFERENCES 

[1] D. \V. Barnes, b'pectral seqvence constructors in algebra and topology, lk|em. Amer. Math. 
Soc. 53 (1985). Mt 86e:55(132 
[2] H. Cartan and S. Eilenberg, Homological Algebra, Princeton University Prcss, 1956. Mt 
17:1040e 
[3] A. Guichardet, Suites spectrales à la Hochschild-Serre pour les produits croisés d'algèbres et 
de groupes, .I. Algebra 235 (200l), 744 765. M1R 2001m: 16013 
[4] H.J. Schneider, Representation theory of Hopf Galozs extensions. Hopf algebras, Israel .. 
Math. 72 (199()), 196 231. Mit 92d:1647 
[5] D. Stefan, Hochschild cohomologg on Hopf Galois extenston...1. Pure Appl. Algebra 103 
(1995), 221-233. MR 96h:16013 

1 LITTLE \VONGA ROAD. CREMORNE NS\V 2(19(. AI'STRAI.IA 
E-mail address: donb@netspace.net, au 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2771-2793 
S 0002-9947(03)03265-3 
Article electronically published on February 25, 2003 

FORMALITY IN AN EQUIVARIANT SETTING 

STEVEN LIIJLYWHITE 

ABSTRACT. ,Ve define and discuss G-formality for certain spaces endowed with 
an action by a compact Lie group. This concept is essentially formality of the 
Borel construct ion of the space in a category of com n|utat ive different lai graded 
algebras over / ---- H*(/3G). These results may be applied in computing the 
equivariant cohomo]ogy of their ]oop spaces. 

1. INTRODIICTION 
In this paper we consider G-Sl)aces and give formality results for them in an 
equivariant category. More si)ecifically, given a G-space 31. we discuss fornml- 
itv of the Borel construction EG x G 3[ or, eqlfivalently, formality of the com- 
plex A(11I) of equivariant differential ferres. However, in the equivariant settiug, 
the mai) 111 -- {pl.} is rei)laced 1)3" EG XG .I -- BG, and consequcntly ail the 
comnmtative differential gra(led algebras involved are naturally R-algehras, where 
R = H ° (BG). Thus formality may be considered in the category of comumtatiçe 
differential graded R-algehras. We shall also consi(ler the augmented case, corre- 
sponding te eqlfivariant base I)oints, which are the saine thing as fixed points of 
the group action. We should like te call a G-space 11I %quivariantly formal'" when 
its Borel construction is formal in the above sense. However, the terre "eqnivari- 
ant formalitv" has corne te be used te describe the degeneration of the spectral 
sequence of the fibration JII -- EG xŒE 11[ -- BG, owing te the i)ervasive influence 
of [11]. Thus xve shall adopt the terminology "G-formal" in this paper. 
We give seine general results concerning G-formaliy of products and wedges and 
reductions te subgroups. This is followed by several examples of G-formal spaces, 
including compact Ki4hler manifolds and formal elliptic spaces, among others. Of 
course, we rnnst make appropriale assumptions on the G-actions of these spaces for 
the results te hold. 
As an application of these results, we compute the equivariant cohomology of 
loop spaces. (If 11I is a G-space, then se is the loop space of 11I in thc obvious way.) 
Our motivation cornes frein considering the cohomology of symplectic quotients of 
loop spaces, sec [18], although the results are of general topological interest. 
shall use an "'equivariant" bar complex te compute the equivariant cohomology 
of the loop space. If the G-space 11I is G-formal, then the bar complex, which 
is generally a double complex, loses a differential and becomes a single complex. 
allowing for seine casier calculations. In the last section we compute an exmnple. 

Received by the editors January 1, 2002. 
2000 Mathernatics Subject Classification. Primary 55P62; Secondary 55N91, 18G55, 57T30. 
Key words and phrases. Rational homotopy theory, equivariant cohomology, bar complexes, 
loop spaces, homotopical algebra. 

@2003 American MathematicaI Society 



2772 STEVEN LILLY%'H ITE 

In an appendix, we discuss bar colnplexes and Eilenberg-Moore theory concern- 
ing the pull-back of a fibration. \Ve also consider equivariant versions of these 
results, which are used in several of the proofs in the lllain body of the paper. 
In what follows, we shall generally assulne that G is a compact. COlmected Lie 
group and that all spaces are colmected. Whenever we need to use the localization 
theoreln iii equivariant coholnology, we shall assulne that the spaces under con- 
sideration are of the holnotopy type of filfite-dimensiolml G-C\V-colnplexes, and 
flu'therlnore that they have finitely lllallv COlmective orbit types, lnealfing that the 
set {[G°] ].r e /I} is filfite, where Gx is the stabilizer subgroup at x. G°x is the 
comected COlnponent of the identity, and [G°] denotes the set of conjugacy classes 
iii G. This latter condition is autolnatically satisfied, bv the way, if ,I is colnpact 
or if G = S 1. 
I would like to extend lny appreciatiol to Chris Alldav. who took the tilne to read 
the lnalmSClipt and offel'ed advice on several key points. In particular. Proposition 
-1.7 is due to hiln. 

"2. /«C)Ç.A AND FORMALIT5 
In this seçtion we recall SOlne ilnt»ortant facts al»out the category of COlmnutative 
differential graded algeblas, the notion of forlnalitv, and the colmectioll with ratio- 
nal holnotopy theory. \V shall assume ibr now that out algebras are /«-algebras, 
where t" is a field of charaeteristic zero. \Ve shall denote l»y/«CT)Ç ° the category of 
COlmnutative diflerential graded /¢-algebras that are concentrated in non-negative 
degrees and have a differential that raises the degree by one. \Ve assume further 
that H°(A)  k, for ail A in hC7)91 '. \Ve shall denote bv /«C7)9 the category 
of algebra8 in /«CT)ÇI ° that are auglnelted over t" (i.e., there exists for each A a 
lnap  " ,4 --/«, with/« COlCentrat.ed in degree zero), together «ith auglnentation- 
preserving lnaps for morphislns. \Ve shall ca]l an object of hCT)ÇI (resp. kCT)Ç.A °) 
a/,'CDGA (resp. kCDGA°). 
We recall Quillen's abstract approach to homotopy theory, [22], [23]. He begins 
by defining the notion of a closed model category. A closed lnodel category is a 
category, C, with 3 distinguished classes of lnorphislns, called cofibrations, fibra- 
tions, and weak equivalences, which satisfv a nulnber of axiolns. The holnotopy 
category, Ho C, is defined to be the localization of C with respect, to the class 
of weak equivalences. Quillen introduces a notion of homotopy and shows that 
Ho C is equivalent to the more concrete category ho C which has for its objects the 
cofibrant/fibrant objects of C, and for its morphisms the homotopy classes of maps. 
We point out the importmlt fact. that two objects X and }" in Ho C are isomorphic 
if and onlv if there exists a chaill (in C) of weak equivalences 

(1) 

In [4] it is shown that the categories kC7?(ï,,4 ° and kCDgI are closed model 
categories where the weak equivalences are the quasi-isolnorphisms (maps that ill- 
duce an isolnorphism on cohomology), fibrations are the surjective morphislns, and 
cofibrations are lnaps that satisfy the following lifting condition: a lnap f is a 



FORMALITY IN AN EQUIYARIANT SETTING 2773 

cofibration if for everv commutative diagram 

with p a fil»ration and weak equivalence, there is a mal» ffoto }" to 1: making the 
diagram commute. (Actually, in [4], the authors do hot assmne that H°(A) 
for all algebras A.  have includcd this assmnption for case of l>resentation, but 
the difference is slight.) 
Given a closed model category, C, with initial object ,, an object B is called 
cqfibrat if the map •  B is a cofibration. B is called a cofibr«mt model for 4 
if B is cofibrmlt and there exists a weak equivalence B + A. It follows ffoto the 
axioms for a closed model category that everv ol»ject in a closed model category 
has a cofibrant model. Moreover, therc are various lifting and homotol>y results 
associated with cofibrant algebras: see [4], section 6. X> mention one here. If 
ç : Bx + B2 is a quasi-isomorl>hism, an<l we have a mat> f : A  Be with A 
cofibram, then there exists a lift f : A + Bt such that çf oe f, where m denotes 
homotol»y. 
Note that L'CÇM is I>ointed with point o]»je<'t L'. The homotopy groups of a 
kCDGA A are defined to be 
def (A/(4)  ) 
nA = H , 
where , = ker, for  : A  k a giveu auglnentation of A. 
If f : B  B is a weak equiva]ence of cofibrant kCDGA's, then f. : 'B 
'B is an isomorphism. Tlms, if we define R'(A) a (B) £r B a cofibrant model 
of A. then fl'(A) is well-defined up to isomorphism. Moreover. if f : Ai  A2 is a 
map of kÇDGA's, then f induces a unique homotopy class of maps f : Bt 
for fixed choices of cofibrmt ttlodels Bi. B2 of .41, -42. respectively. It follows that 
there is a unique mal» f. : H(A1)  H(A2). Thus H is flmctorial, and different 
choices of cofibrant models yield naturally isomorphic such functors. 
In kCÇM, there is a special class of cofibrant models called minimal models. 
A minimal model of an algebra A is defined to be a. cofibrant model, ,  A. 
that is connected (0  k), and such that the induced differential on /(«]2 is 
zero. It can be shown that each algebra in #CÇM has a minimal model, unique up 
to isomorphism. If BI is a path-connected topological si)ace, the (pseudo-dual) 
homotopy groups of AI are dened to be H'(M) d fl,(A.[AI) ) = (,), where 
 is a minimal model for A'(M). Here, A'(M) denotes the Sullivan-de Rham 
complex, which is a QCDGA; see. for example, [3] for the definition. If M is a 
smooth manifo]d, we mav also use the ordinary de Rham complex, taking k to be 
Halperin has explicitly identified the cofibrations (and hence cofibrant objects) 
in #CÇM. Çofibrations me the so-called KS-extensions, and the cofibrant objects 
are the KS-complexes. Since these llotiotls will be important to us, we give their 
Definition 2.1. A map f : A  B of kÇDGA's is said to be a IçS-e'lesion if 
there exists a well-ordered subset  C B  = {a'o}, Sllch that .4 C I (£) 
is an isomorphism of commutative graded algebras, where (£) dent»tes the 



2774 STEVEN LILLYWHITE 

graded comnmtative algebra on the set E, and the map is induced bv f and the 
inclusion of E C/3. Identifying/3 with A  (E). the differential on B satisfies 
(1) dB(a @ 1) = dA(a) @ 1, 
(2) dB(1 
where E< = {x [[ < }. If E also satisfies deg(x) > 0 Vx Ç E, and deg(x) < 
deg(x)  fl < a, then f is called a minimal A'S-extension. If A = k, then we 
replace the word "extension" by the word "complex" in the definition, obtaining 
the notion of KS-conple. (A minimal KS-complex is the smne thing as a minimal 
algebra, defined above.) A minimal KS-extension in which A is also minimal is 
called a A-'minimal A-e¢tension. Note that in a (minimal) KS-extension. A(E) is 
a (minimal) KS-complex, with differential such that ¢  1 :A  A(E) + A(E) is a 
map of kCDGA's, where ¢ is the augmentation of A. Moreover, all of these maps 
mav be ruade compatible with augmentations. 
If A is a kCDGA, then its cohomology, H(A). may be considered to be a kCDGA 
with zero differential. 

Definition 2.2. ,| is said to bc for'mal if A  H(A) in Ho(kCl)Ç.A). 
It is easy to see that this (lcfinition is eqnivalent to the following two. 
Lemlna 2.3. Consider tbe cate.qory kCÇ. The following are equivalent: 
(1) A is formal. 
(2) Theve is a diagram 
A  B  H(A), 
• where the maçs are weak equivalences and B s a coflbrant modelfor A. (In 
particular, we may pick B to be minimal.) 
(3) There is a chain of quasi-isomorphisms 

A *- A1 -- A2 *- "" - A, -- H(A). 

This theory has an important application to rational homotopy theorv. It turns 
out that the homotopy category of rational finite Q-type nilpotent spaces is equiv- 
alent to the homotopy category of the fllll subeategory of QC/PÇM consisting of 
algebras A with HA of finite type, [4]. Thus we may "do'" rational homotopy 
theory in a category of differential graded algebras. As an example, if X is a 
path-commcted, simply-connected topological space of finite Q-type, then there is 
a natural isomorphism 
II'(A°(X))  Ho,n(r.(X) ®Q, Q), 
where A°(X) is the QCDGA of Sullivan-de Rhmn differential fonns on X. If X is a 
smooth manifold, the saine statement for homotopy groups holds if we use instead 
the de Rham algebra A°(X) and replace Q coefiïcients bv NI, or C. There is hot a 
corresponding equivalence of homotopy categories over NI or C, however. 
A path-connected topological space is said to be formal if its Sullivan-de Rham 
algebra A°(X) is fonnal. If X is a smooth manifold, we may use the de Rham 
algebra and real or complex coefficients. However. a well-known result in rational 
homotop.v theory states that formality over NI or Ç implies fonnality over Q: see, 
for exmnple, [15]. 



FORMALITY IN AN EQUIVARIANT SETTING 2775 

Formal spaces include comI)aCt Kfihler mmfifolds and many homogeneous spaces, 
including compact globally synlmetric sl)aces. Produçts, wedges, and Comlected 
smns of formal spaces are agaill forlnai. The top«,logical consequences of formality 
include the vauishing of ail Massey prodnçts. Moreover. the rational homotol)y 
type of suçh a space is deternfined solely 1¢« its cohomology algehra (at least for a 
large class of such spaces). 

3. RC)ç AND G-FORMAIATY 

In this paper, we shall Ie concerned with equivariant versions of standard for- 
mality resuhs. Let G he a compact, COmlected Lie group. Then H'(BC:h) is 
isolnorphic te the hCDGA freelv generated bv a finite number of generators of even 
degree.  shali denote H de H'(BG).  dt, fine the category CÇM ° te be 
the category of comnmtative differential graded -algelras.  shall contimw te 
assume that H°(A)  k for ail algebras A. Tlms, we ohtain a faithflfi forgetfld 
fimctor ff'oto RCÇA  te kCDÇA".  also define RCÇA te l»e the category of 
commutative differential graded R-algelras augmeuted over R. Conlposiug aug- 
mentations with th¢' augmentatiol R  h. we get a faithfifl forgetfid fimctor fiOnl 
RCDÇA te L.CDÇA. 
It is a standard result that if C is a closed model category and B is au object 
of C, then the "'over category" C/B whose o]»jects are nlaps X  B and whose 
lnorl)hisms are commutative squares of the type 

X f • " 
1 1 

may be given the structure of a closed model category with the following definitions. 
Such a morphism in C/B will be called a fil»ration, cofihration, or weak equivalence, 
if the map f : X  I" is such in C. A similar statement holds for the %roder 
category", BC. See [7] for these and other results about closed model categories. 
Thus we see that both RCDç  = RkCDçA  and HCDç = RCDç/H are 
closed model categories. Moreover, the sinlplicial category structure on kCDÇ  
defined in [4], section 5. induces a simplicial category structure on RCDA" and 
HCDÇA in such a way that the results of [4], section 5, suitably modified, hold for 
these categories as well (cf. [22], II.2. proposition 6). Frein this, it follows that 
the homotopy results of [4], section 6, suitably modified, hold for RCDçA ' and 
RCDç as well. 
Definition 3.1.  shall sv that an RCDGA (resp. RCDGA ) A is fowml if 
A  H(A) in Ho(RCDç) (resp. Ho(RCDç)). 
If a filnctor j : C  C t)etxvn two closed model categolies preserves weak 
equivalences, then A"  l" in Ho C iluplies j(X)  j(l') in Ho C. Thus if an 
algebl'a A is formal  an RCDGA, then it is formal as an RCDGA , and  a 
kCDGA, etc. 
Sul)pose a smooth lnanifold BI has a smooth action of a compact Lie group 
G. The equivariant coholnolo&v of I mav be computed bv means of the Cartan 
complex of equivariant differential ferres: A;(M) = ((S*  A'(M)) G, dG) where 
the different.ial, da, is zero on S*, and for o.  A" (M), dGt = de-uitx, o, where 



2776 STEVEN LILLYVHITE 

the {Xi } are fundamental vector fields of the action correspmding to a basis of l- 
and the { u, } are the corresponding algebra generators of S*. which are given degree 
two. If 31 is just a topological space, we llltly compute the equivariant cohomology 
of 3I bv means of the QCDGA A(3I) of [2], when G = S t. Alternatively, ce 
could use thc de Rham algebra of the Borel construction. A'(EG xa M) when M 
is a lnanifold, o1" the Sullivan-de Rham algebra of the Borel construction when M 
is hot a manifold.  shall let A;(M) possibly denote any of the above kCDGA's. 
leaving it to the reader to interpret which mdel one prefers to use,  xvell as whid 
gromM fiekl k. FLra çOlnl)rehensive tleatment of equival'ialt de Rham thcory, see 
[:]. 
Using either model, it is olvious how to oltain an R-algelra structure on A(M). 
i 
If is indu«cd 1,v H  ,t(pt.)  A(M), »vhere the fil'st map is a choice of 
mildlnal model for A(pt.) in kCÇ, and the second map is induced from the 
map 1I  {pl.}. If vc use the Cmtan models, then the algebras A(M) are 
manifestlv augmented (ver H whcn the group action bas a fixed point. This is 
lecause in the Cmtan model, A;(pt.) = , and thc inclusion of a fixed point gives 
a mal» A;(II)  A;(pt.) = . [h,wever. if we use the Sullivan-de ham complex 
ff the Bnel «mstruction. then A(pt.) = A'(BG) ¢ R. Thus we nmst use a quasi- 
isomorphic «omplex that is Slnaller and augmented over . In [1], Allday shows that 
the «Oml,lex ,/-()is quasi-isomorphi« to A;(M), where ,/ A(3I)  A(pt.)is 
induced by the in«lusion of a fixed point into [, and  is embedded in A(pt.) via 
i as above. Clearly. tl-() is augmented over R. and is flmçtorial for eqtfivariant 
maps of G-spaçes. We shall almse notation and continue to write A;(M). even 
when we may really mean I-(R). 
Let ÇTO denote the category of l)ath-comected topological G-spaces with 
morphisms the equivariant maps. Then the under category {pt.}ÇçO consists 
of "%ased G-sl)aces", which is the saine thing as G-spaces with non-empty fixed- 
point set and a choice of base point in the fixed-point set. Then A(-) gives a 
flmctor from ÇçOç to RCDÇ  and from {pt.}ÇçOç to RCDç. 

Definition 3.2. We shall sav that a G-space .I is G-formal if A(3I) is formal 
as an RCDGA". A G-space M with equivariant base point p (i.e., a choice of fixed 
point p  3I a) is G-formal af p if A(3I) is fornlal as an BCDGA. where 4(M) 
is augmented via the inclusion of p into 

If we continue to define a minilnM model of an BCDGA as a connected cofibrant 
model M for whMl the induced diflbrential on ker e/(ker ) is zero, where  is an 
augmentation over . thell there moEv hot be a minimal model for everv algebra 
in C. An exmnple is .ç actiug bv rotations of .q2 about an axis. It is easy 
to see that there can l»e no minimal model for A. (S ) in BCDÇA. However. 
there is a fairly canonical choice of cofibrant model for an CDGA. Let   A 
be an RCDGA . Then the map R  A, viewed in kCÇM, mav be factored 
as     ,M  A with the first map the inclusion, the latter lnap a quasi- 
isomorphism, and M a minimal KS-complex, [14]. Note that the differential on 
R @ M is hot the tensor product differential: see the definition of a KS-complex 
(Definition 2.1). The map R  R G' . is a nfinimal KS-extension. in particular a 
cofibration in kCçM, and hence we sec that R , . is a cofibrant model for A in 
RCÇM . Suppose A is, moreover, an algebra in RCDÇM. and let e : A  R be 
its augmentation. Then composing B ï .  A G B gives an -augmentation for 



FORMALIT' IN AN EQUI\%RIANT ,qETTING 2777 

R  .A4. Thus, R . .,M/ becolnes a cofibrallt model fbr 4 in the categorv RCTbÇ.A. 
As defined, itis unique up te isonorphism. 
Fol" those algebras of the tbrm A(./II) arising ri'oto a group action on the spa«e 
31. this cofibrant model is more explicitly given b) the Grivel-Halperin-Thomas 
theoreln, which states that there is a c(mmmtative diagram 

(z) 

i 
R --* R '.in,..t --, 

asso«iated te the filtration 11I -- EG xc M -- Bd7. where .A4 is a minimal model 
br M. and the ],otom row is a A-minimal A-extension. see [121, [14]. 
Definition 3.3. && shall rebr te R .   the G-mod«l of A, or just simply as 
the G-mod«l of M. when A : A(M). 
Sometimes we mav ehoose te dent,te it lx _( df B. . Note that R , 
may tdl te lin minimal as a k('D(',A. 
Folloving [1], [3], given a lmth-colmectcd G-space 1I with equivarimt I,ase point 
(i.e., a fixed point) p. the equivariant (pseudo-dual) k-lumotopy groups are defilmd 
te 
(3) Hâ,p(M) a '(R ._ Al)= H'(kere/(kel'e)2), 
where ge : R  . + R is the R-algebra augmentati«m induced ])x the inchisiol of 
p into M. as above. The assigmnent (31. p)  (R..ç.«) gives a flmctor ff'oto 
{pl. }çOç te Ho(RCDçA). and t he equivariant pseudo-dual k-homotoW group 
are fimctorial as well. Note that if M is G-forlnal. then the equivaliant pseudo-dual 
k-homotopy groups are determilmd by the equivariant coholnology ring of 1I. 
The following lelmna is uscflil for comparilg the equivariant pseudo-dual k- 
holnotopy groups te the oldinarv pseu&-dual k-homotopy groups of the Borel 
construction. 

Lemlna 3.4. Let A he an RCDGA «td let R. .A4 t,«- the G-model for .4. Then 
R ,. )., is minimal in kCDgA. 

Pro@ We have the augmentatiol g : R ,.-_ a'vl --, R. vhich is a lnap of RCDGA°'s. 
The differential. D. on /7 ._a .A4 satisfies D(r. ' 1) = 0. for r  R, and generally has 
the brln D(1 " oz) = r C 1 4-  ri  ai 4- 1 ._ç da, where a, ci,  .,. r. ri  R with 
deg(a).deg(a).deg(r).deg(r) > 0, and where d is the differential in «VI. New. 
I) = eD(1 ® a) = "r + rie(ai) + e(da). Since da  (+)2. and e is an algebra 
map, it bllows that.  ris(ai) + e(da)  (17+) 2. Hence, we nmst bave that r = 0, 
and it fbllows that 17 .-> .k4 is mininml. [] 

As ait exalnple, the pseudo-dual k-holnotopy groups of the Bolel constrnction of 
S t acting 011 '2 do hot distinguish the trivial action froln a standard non-trivial 
one, whereas the equi'a.riant pseudo-dual /-honlotopy groups do. 

4. C-ENERALITIES CONCERNING (ïLFORMALITY 
In this section we give some basic results about G-formalitv. ilwluding reduction 
to subgroups and the G-brlnalitv of products and wedges. 



2778 STEVEN LILLYçVHITE 

\\ I)egin 1)y noting that formality in the category HCT)ÇA ° is equivalent to for- 
malitv in kCDGA, hl genera.1, for two R-algebras A and B, A  B in Ho(kCÇA) 
does hot imt)ly that A  B in Ho(RCÇA°). Nevertheless, we have the following. 

Lemma 4.1..4s.s'ume that R -- 4 is an RCDGW' and that we give H(A) the 
j* 
R-algcbra structure R -- H(A). Then A is formal in I,'CT?gA if and onlg if A is 
formal in RCDÇ.A ('. 

Proof. If 4 is tbrmal in RCT_GA", then it xvill be so in 1,'CTÇA, as we have noted 
al»ove. Let us now assume that A is formal in/,'CY)ÇA. Let ,'( lin a minimal model 
for A and let R 9 .'v/be the G-modvl for A. Thon we have a «onmmtative diagram 
of/'( DGA s 

(4) 

.4 

Sin«e R, A is «ofibrant in kCI?ÇA, tllcre exists a nlap. which is ne«essarily a quasi- 
isomorphism, H.-o A -- N" making thc uppcr right square homotopy commute. This 
gives us a quasi-isomorphisn  • R O Ad -- .&oe -- H(A). Thon the map 

(5) 

(*)- r)* 
R®,H(A) -« H(R®.Ad)---,H(A) 

is a quasi-isolnorphisln and a mai) of R-algebras. 

R«mark 4.2. \Ve note that this is hot true for lnaps, however. That is, if f • A -- B 
is a nmp of R(DGA s, and fis formal as a mal) of/,'( DGA s. then f need hot be 
a formal map of R(DGA s. 

In the category RcT)gA, fi)rlnality is a. conccl)t distinct fronl formality iii the 
category kCg)ÇA, hl fa«t, it is easy to see that .,I is G-fornml a.t p if and only if 
the map i " BG ---, EG xŒ M is a formal map. where i is the map indu«ed bv the 
in«lusion of p into 3I. 

Definition 4.3. Sui)pose that G acts on a space M. Then the Serre spectral 
sequence associated with the fibration 31 - EG xa M -- BG is the saine as the 
spectral sequence (from E2 onwards) obtained ri'oto the G-lnodel R ® ,/ via the 
filtration ffP = R >-p ® flA. If this spectral sequence degenerates at the E2 terre. 
then [11] refers to 5I as being equivariantly formal. For obvious reasons, we wish 
to avoid this terminology; however, to conform as well to current trends, we shall 
sa3 that 31 is er when this spectral sequence degenerates at the E2 terre. 

Proposition 4.4. Let G act on a space M. Suppose that K C G is a cloaed. 
connected subgroup. If 5I is G-formal at p (or G-formal) and er. then 31 is K- 
formal at p (resp. K-formal). 



FORMALIT IN AN EQUIVAIIIANT SETTING 2779 

Ptvof. We fil'st consider the case where 'I is G-folnml at p. The inclusion h" C G 

induces a pull-lmck diagraln 
Eh xtç 111 -- EG xG 111 
i 
B Iç -- BG 

We shall denote H'(BG) by Re;, a.nd sinfilarlv for Hl. If we are using the 
Cartan complex of equivariant differential forms, then there is no lnobleln with the 
proof. If ce are using Allday's construction, q-l(H), as notated al»ove, then we 
face file possibility that this construction may hot be flmctorial with respect to 
changing the group. This is because there may hot exist choices of minimal models 
so that RG -- Rh -- .t'(BK) conmmtes with Re, -- .I'(BG) - -I'(BIç). Then 
there «ould hot exist an iuduced map q-  (R) -- q- l (/ï't,)- 
This problenl may be circumvented by the following procednre, as pointed out 
to us by C. Allday. ('onsider file diagranl (6). Let f" &,note tlle mal»ping cylindcr 
of the top row, and Y the malpillg cylinder of the bottom row. Then we have a 

in which the nmps jl,fl induce SUljections on differential forms and the maps j2,. 
induce quasi-isonlorI»hisms on differential forms. Il is easv to show that we nmv use 
17- to form the complex q-l(/ï'G), as discussed in section 3. and that this COlnplex 
will be quasi-isomorl»hic to ,4(1I), and G-formal at p if 31 is G-formal at p. 
Moreover, we now may obtain a COlnlmltative diagram 

A'(Y)  A'(BIç) 
in which the vertical arrows are quasi-isomorphisms, since the inap A°(Y) -- 
A°(Btç) is onto. This follows bv the result for RCDg,A. which is the analog of 
By Lemma A.1 of the appendix, there is a qui-isomorphism of kCDGA's 
(9) (A'(Blç),A'(BG),Ab(M))  A-(M), 

where we are abusing notation in the event that we are using Allday's construction. 
Then, in either case, we obtain a quasi-isomorphisln of R/-CDGA's 

(0) 

[(RI«,Ra, A'a(M))  A-(M). 

The bar complex (10) is an R/«-algebra via the R/« factor, and has an RK-augmen- 
ration given by e(rK,ct) = rKï(eG(Ct)), where r/ ¢ Rê,', et ¢ A.(/I), and ea " 



2780 STEçEN LILLh V HITE 

.4;(I) -- Rc is the augmentation of I for the action of G. Bv the asslnp- 
ti«m of G'-formality, we get a conmmting diagram whose vertical arrows are quasi- 

isomorphisms: 

(11) 

Rb  RG -- 
Riç  RG --" H(ll) 

Then xxe obtail, lhe folh)wing sequence «)f lllaps, which are st'en to be R,CDGA 
«luasi-isolmn'l»hislnS i W standard cmnlmrison theorems tbr their associated Eilen- 
ierg-g Ioore spec tral seq,ences: 
(12) B(R,ç. Ra, .4;(.I))  B(R,,. Ru..a(3I))  (R/,. Ha. H;(M)). 
Now the lmr «omplex B(Rt,. RG. ff;(.ll)) has mly lle singlc difl>rential . alld 
comt,utes TorRo (Rb. H(3I)). Since 1I is cf. Hç.(l) is a free RG-module. Hence 
we have thal (Rh,HG,II(3l)). is acvclic in lmr degrees greater than zero. and 
t ho projection to c()h()lllology 
(13) (Hr,, Re, H5(3I)). + ( Rç, Re, 1]:;(31))o 
 H,-(M) 
is an RçCDGA quasi-iSOlnorphism. 
The case where we consider 31 to be G-formal in the category HGCÇ ° is 
similar. 

Corollary 4.5. Let G act on a space 31. Suppose that M is G-formal at p (or 
G-.formal) rmd er. Then 1I is formal in 
Pro@ Just take K to be the i(lentitv sui)gr()ul) iii Proposition 4.4. 
emark 4.6. If we use Remark A.5 of thc apI)elMix, then we can see that a lx-nlodel 
fbr M is given by Ra(Rh. RG,G(l)) : Rb R (I). 
In line with the general thelne of considering maximal tori in compact. COlmected 
Lie groups, we have the following face. which is due to Ç. Alldav. 
space M is of fiite tpe if H'(M) is a finite-dilnensional k-vector space for ail i. 
Proposition 4.7. Let G act on :I. a«td let T C G be a ma.rimal torus. If M is 
G-formal (G-formal at p). then M is T-formal (resp. T-formal af p). Moreover. if 
M is a space offlnite tpe. p  M . and I is T-formal at p. then M is G-formal 
at p. 
Pro@ X can aheadv see tlmt G-forlnal ilnplies T-formal by the proof of Propo- 
sition dA. XX> only need the fact that now HT is a fi'ee Ho-module. which follows 
ri'oto the well-known fact that as Ho-modules. RT  Ho Ç H'(G/T). 
Showing t.hat T-formel al p implies G-formal at p mv be achieved by imitating 
the proof that A  K being forlnal ilnplies A is forlnal, for K an extension field of 
k. whi«h is corollal'V 6.9 of [15].  Olnit the details, bnt mention the setup. First. 



FOIRMALIT IN AN EQUIV4RIANT SETTING 2781 

we see by I{emark 4.6 that a T-model for A(AI) is given bv/r T (-RG .A4c(M) with 
differential 1 C, De, where De is the difl'ereltial fiJr the G-mode| .Me(iii). Thus it 
suffices to show that if/-ff ®ha ,(/11) is formal as au RTCDGA. then .MŒ(M) 
is forlllal as ail RG('DGA. 
It turns out t|lat the coustl'uctions of bigra(led and filtere(l models of the rçlevaut 
algebras, as in [1.5], givc lUodels iu the category HCTÇ4. Thç proof of corollary 
6.9 may bc imitated xvitllout too llltlch difficulty. [] 

Proposition 4.8. Supl,oSe that X and  arc G-spa«es. both of whi«h are G-formal 
(or as.smne X is G-fo'rnml at p and " is G-formol at q). and suplo.e that one 

or both of them is 
diagonal a«tiou of G. 

Proof. The pull-back diagram 

(14) 

G-fin'mal (resp. G-formal at (p. q)) for the 

x --, b,t.} 

gives rise to a lmll-lmck diagram 

EGxG(XxY)  EGXGI 
EG xG X  BG 

(5) 

  O 
Then we o])tail a.li /( DGA qnasi-isomorphism 

by Lemlna A.3 of the apt)endix. If X and ] both have fixed points, then so will 
their product X x }'. In that case, 0 is a quasi-isolnorphisln of RCDGA's bv Lemlna 
A.3 of thc appendix. FurthCllnOlC, 
(17) - " " - " " 
B(A.(X)...a.())  Ai, 
" B(AG(X ), ,({pt.}), ) 
is an RCDGA ° (RCDGA) quasi-isomori)]fiSlU. Since X aud  are G-forum.l. we 
get RCDGA ° (RCDGA) quasi-isomorphisms of bar cmplexes 
(H&(X),R.H&()) 
by standard arguments comparing the associated Eilenberg-Moore spectral sequen- 
CeS. 
Siuce one or both of X. ) is cf. just as in the proof of Proposition 4.4. the bar 
respect to the bar grading, and the ln'ojection to its cohomology is an RCDGA  
(RCDGA) quasi-isomorphism. 



2782 STEVEN LILLYWHITE 

Proposition 4.9. Let X md I" b« G-spaces u,hose fi:red-point sers are non-empty. 
Picking base points p E X G and q  l'G. we may form the wedge X V I" along these 
base points. Theu G acts on A V }'. If X is G-formal ai p and 1" is G-formal at 
q. then X V }" is G-formal al the join of p and q. 
Pro@ Let ex,e denote the augmentations of equivariant differential forms, and 
let ix, if denotc the inclusions of X, 1" into X V Y. Then Mayer-Vietoris gives a 
short exact sequence 
Thus ix + i- induces an isomorplfisn 
R 
Moreover. since ex, say, induces a surjection in cohomology, the associated long 
exact sequence splits into short exact sequences, and thus 
(2O) nS(X V )  n5EE(X ) e bO')" 
Since ç and k are G-formal. we have maps A(X)  a(X)  H(X) which are 
quasi-isomorphisms of RCDGA's, and similarly for I'. So we have a commutative 

diagram whose rows are short exact sequences: 
(21) 
o  Ab(x) ¢ A.(',) --- Ab(X) 

,4c(Y)   --, o 

Then we obtain maps between the associated long exact sequences in cohomology. 
By the 5-1emma, it follows that the maps 
(22) A,(X) eR A,(Y) 
are quasi-isomorphisms. It, is easy to check that these maps are compatible with 
augmentations and the R-algebra structure, so are RCDGA quasi-isomorphisms. 

5. EXAMPLES OF G-FORMAL SPACES 
In this section we give some examples of G-formal spaces. 
5.1. Compact Kihler lnanifolds. Let 3I be a compact Khler mmfifold, and 
G a compact, connected Lie group acting on 3I by holomorphic transformations. 
 introduce equivariant holomorphic cohomology groups. Since M is a complex 
manifold, the complex-valued differential forms on M are bigraded in the usual 
way. ë shall denote Sg* @ C by simply Sg*. Then we define the eqtfivariant 
Dolbeault. cohomology to be the cohomology of the complex 
(23) ([$9" Ce AP"(I)] ° ;  +  ,z,). 
Here Zi is the holomorphic vector field on M which cornes about by splitting the 
fundamental vector field Xi = Zi + Zi into its holomorphic and anti-holomorphic 



FO1R.MALITh IN AN EQUIVAIIANT SETTING 2783 

colnponents. The gellerators ui E Sg* are given bidegree (1.1). The operatol's act 
in a Silllilar way as for file ordinarv equivariallt cohonlology. \\k' shall denote the 
qth cohollology of this complex by H'q(AI). 
The following theoreln was proved in [17] and ildependently established in 
Theorem .1. Suppo.s'e that Al is a compact KShler maifold edowed with a 
holowophic action of a compact, covected Lie 9roup G. and sui»pose that I s ef 
for the action of G. Then M is G-Jbrmal. IJ" M a 
fized point. 
Pro@ The Cal'tan COlnplex is (A(M),da)= ([S* N.4"(3I:C)] 
Let Xi = Zi + Zi be the splitting of the flmdalneltal vector field Xi into its 
hololnoll»hic and anti-hololnorphic parts. The differential d = O + 0 also splits. 
Hence we lnay split the equivariant differential as d +  utx, = (0 +  tz,) + 
(0 +  ut,). The COlnplex [Sg*@ A(M;C)] a is bigradcd by giving 
bidegree (1.1). and taking the usual l»igl'ading on A* (1I; Ç). It is easy to show that 
(24) ([Sg*A'"(M:C)]a ; (+u,z,),(O+u)) 
is a first qua.drant double çOlnplex. Accoldingly we have two canolfical filtl'ations of 
this COlnl)lex.  claire that the spectral sequences COl-lespondilg to both of theln 
degenel'ate at the E1 terln, and lllOl'çover are n-opposite, memfing that 'F p 
H' for p + q- 1 = tt. Formalitv for 4(M) then follows owing to the results in [6], 
sections 5 and 6. 
Let us consider the filtration iii which we take  +  uitz, coholnology first. 
This is the Dolbeault equivarimt coholnology defined above. It itself fOl'III 3 fil'St 
quadrant double complex with the two differentials  and  uitz,. Let us filter 
so that we take the 0 coholnology filSt. Then the E terin for the equivariant 
Dolbeault COlnplex is (additively) 
(25) H([S*  4"(M)]a:O)  [H(Sg*  A'(M):O)] a  (Sg   H(M)) a 
(26)  (SO*) a  H3(M)  H'(BG)  HO(M ). 
Now bv ordinary Hodge thcory fbr COlnpact Khler manifolds, this lt is isolnorphic 
to H'(BG) H'(I). But now there can be no further non-trivial differentials 
iii the spectral sequence, by the aSSUlnption that M is ef. This result follows 
mmlogously for the other filtration, which is just the COlnplex conjugate of this one. 
Furtherlnore, it is easy to see that the two filtrations m'e n-opposite. 
Hence we have a "OGOG-lelnlna'" for the equivariant differential forlns, where we 
mean by 0G the equivariant Dolbeault opcrator as defined above. Forlnality follows 
via the sequence of CCDGA quasi-isomorphisms 
(27) Ab(M)  ker(Oa)  Ha(M ), 
which are the inclusion and projection, respectively. These maps are lnaps of R- 
algebras, and moreover, it follows that for equivariant holomorphic maps between 
M and N, we get a commutative diagraln linking the sequence (27) for .I to the 
analogous sequence for N. In particular, if the action of G on M bas fixed points. 
then the inclusion of one (chosen as an equivariant base point) gives augmentations 
so that the sequence (27) comlnutes with auglnentations. That is, 3I is G-formal 
in RCÇA. 



2784 STEVEN LILLY\VHITE 

Corollary 5.2. Sttppose that M is a compact K5hler manifold edowed witb a 
holomorphic action qf a compact, cone«ted Lie 9roup G. Assume that AI G # 
7Jen M is Gçformal at ang fi.red point. 
Pro@ Lel p G 3I . Let T c G 1)c a maximal torus. Then 3I  ¢ , and a theorem 
of Blanchal'(t savs that ;I is ef for the action of T: sec [9], Cai)ter XII. theoreln 
6.2. By Theorem 5.1. M is T-fi»rlnal at p. By Pl-opositiol 4.7. 31 is G-formal at 
p. 
Rcmark 5.3. The proof of Th«or,ln 5.1 implies an cquival'iant Hodge decomposition 
5.2. EIHptc spaces.  re«all tiret an elliptic space 1I   stmce su«h tlmt both 
tl'(M: k) and I" are finte-(limensonM k-vector spaces, whcre (M) = (I') is 
mhfima] mdel fi»r 1I. b shall nse the followh remflt of [19]. 
PropoMt}on 5.4 (Lui»ton). Let F  E   e  fibr«dion i which F s formd 
«nd «lliplic. and  is .fovmol che simpl.q-conne«ted. If th« Serre spech'al 
of the fihralion degenerot«s «t the E2 l«rm. the E is formd eIso. 
Theorem 5.5. Let 3I e az dlipti« G-poce If M is formel and er. the M is 
(;çformol. If ll G  . lhe zI is G-formol ci ez.q .red poinl. 
Pro@ We lmx-e the filtration ,I  SG XG :11  BG. Thon Proposition SA 
hnpfies t, lmt A.(,ll) i fin'mal as a k('DGA. çhe proof of Lupton's proposition works 
(a(iapthg to oto" situation) Lv finding a model for A(AI) of the form R.,ç that is 
bigraded a a kCDGA. Hcre, ç is the bgraded (nfinhnal) model of 3I. Elements 
of R arc in degrec zero for the second grading, so tha (R  -)0 =  - (,)0- It 
is shovn that wth respect to tlw second grading we have H+(H  ,) = 0. and 
hence the projection to cohomology 
(29) R., M  (R.)o  H(3I) 
is a quasi-isomorphism. Çlem'lv. this is a map of R-algebras. Ioreover. if 3I   
then this mat» commutes with the augmentations over R. This follows because first 
the mal» R ;-)  (R :-))0 commutes with augmentations. Second. since the 
augmentation e : R -   R is a map of RÇDGA's, e(do) = ) for ail . so that 
the map (R , ,)0  H(AI) commutes with augmentations. 
Corollary 5.6. Let 31 be at elliptie space. Suppose that a tortt T acts on 3I u,ith 
3I T  . Sui»pose fltrther that one of the «ompo#e#ls of the ç.red-pomt set, 
Pro@ Since AI is elliptic, it. follows (via localization and loca]ization for equivari- 
ant rational homotopy [3]) that each component of the fixed-point set is elliptic and 
k(AI) = (AI), where k is the homotopy Euler characteristic. But Halperin 
h shown that for elliptic spaces the conditions H °dd = 0 and k = 0 are equiva- 
lent. and moreover such spaces are forma/. Thus 0 = (3I) = (I). Hence 31 
is formal and H°dd(Al) = O. But this latter condition implies that 31 is ef. So we 
mav apply Theorem 5.5. 
Remark 5.7. Suppose G aets on a simply-connected space AI with non-empty fixed- 
point set. Then bv picking a base point in the fixed-point set, we obtain an action 



FOItMALITY IN AN EQUIVARIANT SETTING 2785 

of G Oll the Sl)ace of |,ased loops in 1I, deuoted tM. Since the cohomology of Lll 
is fret, we see that t]l will be G-formal if t/l is er. (Lupton's proof could be 
extended to this case, as well.) If G = T is a torus, and M is elliptic, then the 
condition that t]ll is er is equivalent to the G-model R ).A4(M) being mininlal in 
the category RCDçA: see [3]. 3.3.15. 

5.3. Miscellanea. Next we shall give a few extra examples of G-fornlality. 

Theoreln 5.8, Let 11 be a space u,itb minimal modcl « = /(I'). Sui»pose that 
d.r = 0 .for all x G I "'v«' such th(d deg(.r) < diln lI. Sui)pose fltrther that the civle 
S  = T acts on M. that M i.s @ and lhal cach comportent o.f the fixed-poinl set is 
formal ad satisfics H°ad(jl T) = O. Then ll is T-formal al an9 fixed point. 

Pvof. Since M is ef. the Serre spectral sequence for the fibration AI -- ETXT3I -- 
BT degenerates at the E2 terre. (Note that by the localization the«weln, this implies 
that M w ¢ .) Bv a standard change of basis argmuent, we lllay assllllle that in 
the T-mo(lel (R.,'t//, D) wc havc D.r = (). for x  I 'w such that deg(.r) < (lira 31. 
Let i : ll r '-- M &,note the inclusiou of the fi'<cd-l>fint set. Th('n wc have maps 
of/ï'('DGA's (actually, the algebras on the righ-hand side of the diagram d<) no 
satisy H ° = k, lmt t]tis will m)t l)resent any l)robh'lnS) 

(30) 

i 
A.(I) AI(M r) 
R ® cM(M)  R® 3A(3I r) 
/4(M) HT-(M r) t e t/'(M r) 

whel'e/t is a quasi-isonlorphisnl sinçe 3I T is folmal. Since 3I is ef. the nlap i* is 
an injection. We claire that hi(R ®.A4(M)) ç i*(H-(M)). Since the maps are 
algebra maps. it suffices to check this on alget)l'a genelatol's. Since M is er. the 
localization theorenl shows that i* is an isolnolphisln in degrees >_ dira M. Also if 
c  R-,M(3I) bas odd degree, then bi(o) = [k since H-aa(M r) = 0 bv assmnption. 
So it sutfices to check the claire on algebra generators of R N,/(,àl) of even degree 
less than dira M. Let o be such a generator. If o G R, theu the claire is obviouslv 
true. If o' G .A4(]l), then by assunlption Dc = 0. Then bi(o) = [i(cQ] = j([o]). 
Thus we have a map 

(31) 

which is a quasi-isomorphisn of RCDGA's. 

Corollary 5.9. Let ]I be a smtplg-connected space witk minimol model ](I/). 
Suppose that dx = 0 for all x Ç V « such that deg(x) < dira M. Suppose furtker 
that a torus T acts on M. that M is @ ad that each composent of the fi.red-poivt 
set is formal 



2786 STEVEN LILLYVCHITE 

Pro@ First of all, there is a subcircle S t C T such that 11I si 
of this circle ,S 'l  T induçes a imll-ba«k diagram: 

= AI r. The inclusion 

ES  X s 
B,ç I -- BT 
Since the action of T is er, the Serre spectral sequence for the fibration on the 
right degenerates at the E2 terre. Bul then the saine is true for the pull-back 
fibration. Hence lhe S  action is er  well. Now the result follows from Theorem 
5.8 and Corollary 4.5. 
Corollary 5.10. Let 3I 4 be a .sp,,ce such that Haa(kI) = 0 atd dimkl = 4. 
Eq»pose that a circle S  = T acts on M. Ten M is T-Jbrmal al ang fi«ed point. 
Pro@  have that H°aa(I) = 0, so that 31 is er. Then ai T  . Bv localization. 
H°aa(5l r) = 0. But path-comccted spaces xvith H  = 0 of dimension less than or 
(,qual to 4 are tbrmal; so each component of 3I y is formal. The result follows bv 
Thcorcm 5.8. 
Remark 5.11. A simple example of an Sl-space satisfying the conditions of Theorcm 
5.8, but which is hot Kfihler or elliptic, is the following. Let S  act on S 4 so that 
the fixed-point set consists of two isolated points. Exlend this to a diagonal action 
of S t on S 4 x S 4. Then, removing a ncighborhood of a fixed point, we may forln 
the coimected sure S 4 x $4S 4 X S 4. This manifold then inherits an S  action with 
6 isolated fixed points. It is hot elliptic, and not even synlplectic, since H  = 0. 
It is easy to check that it satisfies the conditions of Theorem 5.8. so is sl-formal. 
(This can also be seen by proving that the commcted sure (ruade in an equivariant 
setting) of G-formal spaces is again G-formal. which we have omitted.) 
 conclude this section with two examples that do hot involve the condition of 
3I being er. 
Lemma 5.12. Let M be a simply-co,nected compact manifold. Suppose that G 
acts freely on M ad dira G k dira I - 6. Then M zs G-formal. 
Pro@ Since G acts freely, M/G is a simply-comected manifold of dimension 6 or 
less. Hence M/G is formal [21]. So EG 
Remark 5.13. Suppose, iii the situation of Lemma 5.12, we have that dira M - 6 
rank(G). Let T C G be a llaXillml torus. Then by Proposition 4.7. MIT is a 
simply-commcted manifold of dimension greater than 6 which is formal. 
Lemma 5.14. Let M be a simply-connected elliptic space. Suppose that G acts al- 
most Jkeely on M (meaning ail isotropy groups are finite), and rank(G) = 
-X(M). Then M is G-fomal. 
Pro@ Since M and BG have finite-dimensional pseudo-dual rational homotopy, so 
does EG x a M, as mav be seen by considering the fibration M  EG x a 3I  BG. 
Since G acts almost freely, H" (EG x aM) is finite-dilnensiolml as well. Furtherlnore, 
(33) (EG xa M) = X(M) + x(BG) = -rank(G) + ra.nk(a) = 0. 
Thus EG x a M is elliptic with  = 0, so is formal. 



FORMALITY IN AN EQUIVARIANT SETTING 2787 

6. AN APPLICATION 

In this section we give an application of G-formality. We will shov« that the 
conqmtation of the equivariant cohomology of loop spaces simplifies considerably 
when the spa«e is G-fin'mal. 
Let us consider a silnply-commcted sImce 31. Suppose that G acts on 31 with 
non-empty fixed-point set. Let p  iii c; tre a choice of base point. Then we get an 
action of G on the loops iii I based at p, (M:p), which we shall oftell abbreviate 
as LI. Let P(ll:p) he the space of paîhs in M, based at p. Then we have the 

fitwation 

(34) 

Lll -- P(M:p) 

where rr is the lna I) sending a path "y(t) to its value at tilne 1, '3,(1). rhleover, the 
G-action induces a imll-back diagraln of fibrations 

(35) 

EG xc Lll --, EG xc P(Al:p) 
BG -- EG xo 31 

Hence there is a quasi-isomorphisln of/?('DGA's 
bv Lemlna A.a of îhe appendix. Now the inclusion of {p} into P(M:p) followed 
by w is îhe inclusioll of {p} into M. These nlaps are equivariant, so induce their 
analogs Oll the Borel constructions. Hence we get an ÇDGA quasi-isomorphism 
(37) (. «b(*),) 
Proposition 6.1. Let g act o a simply-coectcd s'pace .I with non-empty fixcd- 
point set. so thot G acts on M. Suppose tbat I is G-formal. Then there is an 
isouorph ism of R-algebras 
(38) H(.I)  TorZ,(M)(. ). 
Pro@  bave that .45(Lll ) is quasi-isomorphic to (B,.45(M),B ) (via a se- 
quence of BCDGA quasi-isomorphisms). The ssumption of G-forlnality mcans we 

have a comnmting diagram of R-algebras 
(39) R , « 
, « HS(h 
 obtain RCDGA quasi-isolnorphislnS 
(40) 

Aa(M ) « 

b(R, Ab(M),R ) ,- (R, A4¢E(IU),R)-- 



27t8 STEVEN LILLYV'HITE 

This follows bv stalidald comparisol theorelns for the Eilenberg-Moore spectral 
sequences a.ssociated fo the bal" complexes. Thus we have that Y(R. H,,Ill). Ri 
is quasi-isolnorphic to A_;(fLI) (via a sequence of/?ÇDGA quasi-isolnorphisms). 
But the cohomology of B(R. H.(M), R) is TorH;(al)(R, R). [] 
Re'm«u'k 6.2. We can alwa..vs choose any lesolutiol fo compute Tor. But we note 
that we may always use the bar resolution, and using Lemma A.4 of the appendix. 
we see that when ;'I is G-f(»rlnal. H;(tII) mav be COlnputed via the (single) 
COlnplex 
(41) (/gR(H. H5(3I), R; 6). 
flemark 6.3. We conld also ol)tain analogous lesults fol' the equivariant cohomology 
of the free loop sl»a('e L,1l. 

ï. AN EXAMPLF 
|n this section we COlnpute ail exalnl)le of le equivariant cohomology of tle 
»ase(1-h)ol) sl)ace using fle l(n'lnalize(1 bar c«nnl)lex over  of elnark 6.2. 
7.1. Example: S 1 aeting on ILq . The circle .b ' acts on the 2-sphere .b " by 
rotations about an axis. sav the z-axis when ,5 '2 is the unit sphere in N a. This 
action is hololnOrl»hic and Halniltonian. Thus by Theorem 5.1..q.2 is G-forlnal 
(G = ,5q). It is easy to show that the equivariant coholnology ring is 
(2) , (S: »)  ['. ,]/( + 0(.- ,,), 
where the degree of .r and u is t, and R = k[u] acts as multiplication bv u. 
The fixed-point set, F, consists of the norfl and south poles, kk shall write 
F = {N,S}. Let lS u be loops based at the north pole. Then S  acts on !S . 
Then the equivariant cohomology of the based loops, Hs, (lH). lnav be computed 
as the cohomology of the bar coml)lex 
(43) ([(»[-], (.«_ ,,). []) ; 
Let w 1)e the SVlnl)lectic form on S 2. Then .r is represented - the form w,- 
uf G A.,(S2), and u is rel)l'esent('d by the forln u G A.,(S2), using the (artal 
COlnI)lex of equivariant differential forlns. Hem. f is the molnent lnap which sends 
a l)oint on S 2 C N3 to its z-colnponent. Then the inclusion of the north pole {Y} 
into S 2 induces the auglnentatiol, H.(S 2)  H}({N}) k[u,] sending x  -u 
and u  u. k omit the details of COlnputing the bar complex, but one finds 
without difficulty the cohomology generators (1. x. .... x. 1) in degree n for n odd. 
and (u /2. 1) in degree n for t even. Owing to the shuffie product structure on the 
bar complex, onc sees that. as an R-algebra. 
whel'e a'i is an indeterlnilmnt of degree i. 
Remar 7.1. In this examI)le, the lOllnalized bar COlnplex (R. H., (S). R) is 
actnallv isomOlphic to the k('DCA minimal model for £S  x s, [2j2. which is 
(45) .s, ., s = A(..,...) (& = 0: &. @ = 



FORMALITY IN AN EQUIARIANT SETTING 2789 
where the degrees of u and p are 2. and the degree of .r is 1. The isomorphism is 
given by (1,x, 1)  .r, (u, 1)  u. and (l.r,.r, 1)  y. 
e'ntar" 7.2. In this example, the space ES 1 xs t]S 2 is hot fornml, implying that 
tS 2 is hot G-formal. hMeed. Massey products alomM. 

APPENDIX A. [AR COMPLEXES AND EILENBERG-IIOORE THEORY 

In this aplwndix we shall discuss the theorv of Eih,nberg and Moore concerning 
tmll-backs of fibrations.  will also consider equivariant versions of these results. 
For references, sep [20], [241, or [8]. 
Let us sut)pose that we have a filnation F + E  B and a map f • X  B, so 
that we obtain a pull-back diagram: 

(46) 

Then t he maps f* and p* lnake A ° ( X ) and .4 ° (E') (difl'erent ial graded ) modules over 
.4°(/3). Let us aSSulne that B is simply-come«ed. Then a theorem of Eilenberg 
and Moore asserts that there is an isomorphism 

(47) 

0" TOl'Ao(B)(A°(X), A°(E))  H°(Ef). 

We mav use the bar resolution to obtain a resolution of. say, A°(X) bv A°(B) - 
modules. Since we are considering A°(-) to be the de Rham or Sullivan-de Rham 
complex, we will use Chen's normalized bar resolution, see [5] or [10]. 
More specifically, the bar complex is 

(48) 

X 
B(A'(X),A'(B),A'(E)) = 0 A'(X) ® (sA'(B))  ® A°(E), 

where the tensor products are over the ground field k, and s denotes the suspension 
functor on graded vector spaces that lowers the degree by one. Hence the degree 
of an e|ement (a,w ..... v,fl) is deg(a) + }-'î (deg(vi) - 1) + deg(fl), where 
c  A°(X), a:i  A°(B), and fl  A°(E). Actua||y, the bru comp|ex is bigraded. 
We introduce the bar degree, denoted B(A°(X), A°(B), A°(E))o. The bar degree 
of an element (a,a: ..... a,',/3) is defined to be -k. The other grading is the 
normal tenso," product grading, the degree of ail element (ca. ' ..... '., d) heing 
k 
deg(a) + i= deg(cvi) + deg(fl). 



2790 STEVEN LILLYWHITE 

There are two differentials 
(49) d(a. c0,..., c0k, 3) = 

+ 
(.50) -ri(a. ..... ».) = 

of total degree +1: 
(c/a. 1 .... , k,/) 
/,. 
+ OE(- 1)ei-'+l (0¢. c01 ..... cOi-l.d:i.oei+l ..... 
(-)(a.«. .,. 
k-1 
+ (-)«-'+(. ..... 

where ei = degc + (legw +---+ degwi - i. The differential 5 has degree +1 with 
respect to the lal grading, while the differential d bas degree +1 with respect to the 
tensor produçt grading. ()ne mav verifv that da + ad = 0. and we put D de___f d -F ( 
to be the total differential. With the given bigradilg, we get a double complex 
with the two difl'erelitials d aml ri, which giw,s rise to the Eilenberg-Ikloore spectral 
sequence. 
('hen's norlnalized version of this bar eoln|lex is the following. If f ff 4°(B), let 
,5'i(f) l,e the opelator on B(A'(X). 4"(B), .t'(E)) defined bv 
(51) Si(f)(a. COl ..... la)k,/ ) ---- (tq, kU 1 ..... kUi__ 1. f., ..... 
for 1 <_ i _< /« + 1. Let II be the subspace of B(A'(X), A'(B), A'(E)) generated 
bv the images of Si(f) and DSi(f) - Si(f)D. Then define 
(52) (A'(X),A'(B),A'(E)) dej B(A'(.\').A'(B),A'(E))/II. 
Then I| is closed under D, and when H°(B) = I« (B is connected), then I1- is 
acyclic, so that /)(A'(X), 4"(B), A'(E)) is quasi-isomorIhic to B(A'(X), A'(B). 
A'(E)). Notice that in the normalized bar con@ex there are no elements of neg- 
ative degree, and with out assumption that /3 is simply-commcted, we are assured 
convergence of the associated Eilenberg-hloore spectral sequence. The map 0 men- 
tioned above is induced by the ma l) 
(53) 0- B(A'(X), A'(B),A'(E))--, A'(E/), 
which sends all tensor products t o zero except fol" A ° (X)®k A ° (E), where the map 
is (a. [3) /5"0 A f*3. Note that 0(lI') = 0. so that we get an induced lnap 
(54) 0"/(A°(X), A°(B). 4°(E))-- A°(E.). 
The nornlalized bar complex may also be auglnented. The augmentation, e, 
nlai)s all elements of positive total degree to zero. The elements of degree zero 
have the form (f. g), where f ¢ 4°(X) and g ¢ A°(E). Then we define e(f,g) = 
ex(f)eoe(g) = f(xo)g(eo), where x0 and e0 are chosen base points in X and E, 
respectively, and ex,ce are the augnmntations of 4°(X), A°(E), respectively. If 
we choose base points so that the pull-back diagram above preserves all base points. 
then 0 is an augmentation-preserving map. 
The bar conlplex has a natural coalgebra structure. Since we are inputting 
kCDGA's to the bar complex, we also obtain a structure of kCDGA on the bar 
complex via the shuffte product. 



FORMALITY IN AN EQUIVARIANT SETTING 2791 

More specifically, if (et 1 ..... (gp) and (bi,..., bq) are two ordered sets, then a shuf- 
fie r of (al ..... Op) with (bi ..... bq) is a pernmtation of the ordered set (al ..... 
bi,..., bq) that preserves the order of the ai's as well as the order of the ba-'s. That 
is, we delnand that if i < j, then a(ai) < a(aj) a,,a «(bi) < «(b¢). 
We obtain a product Oll B{A'(X), A°(B),A°(E)) by first taking the normal 
tensor product on the A ° (X)® A ° (E) factors, then taking the tensor product of this 
product with t.he shufl3e product on the A°(B) ®i factors. As usual, we introduce 
a sign (--1) deg(a)deg(/) whenever o is moved past/3. One che«ks that this product 
induces a product ol, Chen's norumlized con,plex,/)(A" (X),. 1 ° (B), A" (E)). as well. 
Thus we arrive a, t|le following lmnma, whosc proof is left to the reader. For more 
details, see [16]. 

Lemma A.1. A.'surne that we bave the pull-back diagvam (46), wh«re p is a fibva- 
tzon and 13 is sirnply-connected. Then the nornalized bar «omple.c 

is a kCDGA. Moveover, 

0 : O(A'(X). A'(B), A'(E)) -- 

is a quasi-isomorphism of kCDGA "s. 

Remark A.2. We note that Chcn's normalization is flmctorial. That is. if we haxe 

a conmmtative diagram of kCDGA's 

then we get a map of kCDGA's/)(A, B1,C1) --,/)(Ae. B2,C2). 

We may fornmlate an equivariant analog of the bar complex. Let us consider 
again the pull-back diagram (46). If we suppose flrther that X.B. and E are G- 
spaces, and that f and p are equivariant maps. then we obtain a pull-back diagram 

(56) 

EGxX ï , EGxB 

Note that we are assuming B to be simply-connected, which in turn implies that 
EG xo B is simply-connected as well. We ma- apply Lemma A.1 to the diagrmn 
(56). However, the bar complex (A(X), A(B), Ab(E)) has the extra structure 
of an RCDGA ° or RCDGA, depending on fixed points. We mav give it an R- 
algebra strucure via the R-algebra structure on the Ab(X ) factor, and we define the 
attgmentation as above, assmning that we tan choose our base points as described 
before to be actually fixed points of the group action. We arrive at the following. 
Lemma A.3. Assume that in the pull-back diagram (46), we have that X, B. and 
E are ail G-spaces wilh f andp equivariant maps. Then the normalized bar complex 
(A(.'(),.4c,(),.%(E)) 



2792 STEVEN LILLYVHITE 

is an RCDGA °. Movcover, 

is a quasi-isomorpbism of RCDGA °'s. If we asu'me further that all fixed-poznt 
.ets are "on-e'mpty. and the diagram (46) preserves base pohts chosen from the 
various fixcd-pont sct., the te normdizcd bar complex is at RCDGA. and 0 s 
a qua.i-isonorphism of RCDGA "s. 

In this equivariant case, we lnay flnthcr simplify the bar COlnplex, following an 
idca of [10]. Let us consider the bar COlnplex over R: 

(57) 

s, (A5EE (X), AS(S),-«5E(ï)) = ( A5EE(X) c,. (.-5 (S))® «) 5(:). 
t-0 

where ail thc tensor products are over 

Lemlna A.4. Sqqose tbat ,4.13. avd C are RCDGA "s and we bave .morpbisms 
of R('DGA "s ,4 -- B - C, wbeve R = H°(BG) for (7, a compact, connected Lie 
gvow. (II'e use this scquctce to defiw a (differetial gvaded) B-module structure 
o' A ad C) &ppose, flwtber, eitbev tbat for each r  R. r is ot a zevo-divisor 
ir A. or tbat this conditio't bolds for C. Tbev tbe natural projection 

(58) 

t.(A, t. C)  t.A, g, c) 

is a quasi-isomorpbisrn of RCDGA 's. 

Pro@ We have that Bn(A. 13, C) = Ba(A. 13. C)/V, where 1" is the sub-complex 
generated by all elements of the form 

(59) 

(a. bi ..... rb ..... bk, C) -- (a. bi,..., rb+ ..... ba, c). 

where r  R,a  A,b i  13. and c  C. It is due to the fact that all elements of R 
bave even degree that l: is closed under the differential D = d + 5. We clailn that 
V is, iii fact, acyclic. To see this. consider the lnap s • l "i - l-i-1 defined bv 

(60) s{(o,b I ..... ïl)i,... ,bh.c) -- (a. bl .... ,'rbi+ .... ,ba.c)} 
= (-1)  {(a, bi ..... rbi. 1. bi+l,..., bk, c) - (a. bi, .... bi, r, bi+l ..... bk, c)}, 

where ai = deg a + deg w +- • • + deg cri - i. It is straight forward but t edious to check 
that ris + sd = 0. and that 5s + s5 = id., so that Ds + sD = id., and consequently 
V is acyclic. Moreover. it is easy to check that l" is an ideal, so that the product 
on thc bar COlnplex induces a product 011 the bar complex over R. [] 

Remark A.5. Lemlna A.4 is valid using the normalized bar complex. 

Corollary A.6. In the situation of Lemma .4.3. 

(6) 

O'n(A.(X),A.(B),A.(E))--, Ab(E.f ) 

is a quasi-isomorpbisn o.f RCDGA  "s (RCDGA "s). 



FOF{IIALIT IN AN EQUI'vAF{IANT SETTING 2793 

R EFERENCES 

1. C. Allday, Rational homotopy and torus actions, Houston .hmrnal of Math. 5 (1979), 1-19. 
IlR 80rn:57033 
'» , Invariant Sullivan-de Rbam forms on cyclic sers, .!. London Math. Soc. 57 (1998), 
478 490. NIR 99g:55015 
3. C. Allday and V. Puppe. Cohomological metbods in transformation groups, Cambridge Uni- 
versity Press, 1993. hIR 94g:55009 
4. A. K. Bousfield and V.K.A.M. (;ugenheim, On PL de Rham theory and rational homotopy 
type, Memoirs of t he American Mathematical Society, no. 179, 1976. MH 54:13906 
5. K. T. Chcn, R«duced bar constructions on de Rham complexes, Algebra, Topology, and Ct- 
egory Theory, pp. 19-32. Academic Press, New York, 1976. hIR 54:1272 
6. P. lï)eligne, P. Grilths, .l. Morgan, and D. Sullivan, Real homotopy theory of Kiihler manifolds, 
Invent. Math. 29 (1975), 245 275. MR 52:3584 
7. \V. G. Dwyer and .l. Spalinski, Homotopy theories and model categomes, Handbook of Alge- 
braic Topology (f.M. Jaunes, ed.), Elsevier Science B.V., 1995, pp. 73-126. MR 96h:55014 
8. S. Eileuberg and J. C Moore, Homology and fibrations I. Coalgebras. cotensor product and 
its derived functors. Cmment. Ma.th. Helvetica 40 (1.q66), 1.q9 236. hlR 34:3579 
9. A. Borel, Seminar on transformation groups, Ann. of Math. Sudics, vol..16. Princeton ['ni- 
versity Press, 19t50. lkIR 22:7129 
10. E, Getzlcr, J. lï). S..loues, and S. Petrack, Differential forms on loop spaces and the cyclic 
bar complex, Topology 30 (1.q91), 339-371. lkI 92i:58179 
11. M. Goresky, R. Kottwitz, and R. NIacPherson, Equzvariant cohomology, hoszul duality, and 
the localization theorem, Invent. Math. 131 (1998), 25-83. MR 99c:55009 
12. P.-P. Grivel. Formes différentielles et suites spectrales, Ann. lust. Fourier 29 (1979). 17-37. 
MR 81b:55041 
13. V. Guillemi, and S. Sternberg, Supersymmetry and equivariant de Rham theory, Springer- 
Verlag, New York, 19.q9. lkIR 2001i:531,10 
14. S. Halperin, Lectures on mtnimal models, Méro. Soc. ikIath. France, vols. 9-10. 1983. 
85i:55009 
15. S. Halperin and .1. Stasheff, Obstructions fo homotopy equivalences, Advances in Mathematics 
32 (1979), 233-2ï9. hIR 80j:55016 
16. S. Lillywhite, Bar complexes and formality of pull-backs. Preprint. 
17. __, The topology of symplectic quotients of loop spaces, Ph.D. thesis, The University of 
Maryland. College Park, NID.. 1998. 
18. __. The topology of the moduli space of arc-length parametrised closed curves in Euclidean 
space, Topology 39 (2000), 487-494. hlR 2001b:55028 
19. G. Lupton, Variations on a con2ecture of Halpemn, Homotopy and Geometry, vol. 45, Banach 
Center Publications, 1998, pp. 115-135. hlR 99m:55014 
20. J. McCleary, User's guide to spectral sequences, Publish or Perish, luc., V'ilmington, DE, 
1985. hIR 87f:55014 
21. T. J. Millet, On the formality of (k - 1)-connected compact manifolds of dimension less than 
or equal to 4k - 2. Illinois Journal of Math. 23 (1979), 253-258. hIR 80j:55017 
22. D. G. Quillen, Homotopical algebra, Lecture Notes in Mathematics, vol. 43, Springer-Verlag, 
1967. hIR 36:6480 
23. __, Ratzonal homotopy theory, Ann. of Math. 90 (1969), 205-295. MR 41:2678 
24. L. Smith. Homological algebra and the Eilenberg-Moore spectral sequence, Trans. Amer. Mat h. 
Soc. 129 (1967), 58-93. hIR 35:7337 
25. C. Teleman, The quantization conjecture revisited, Ann. of Math 152 (2000), 1-43. 
2002d: 14073 

DEPARTMENT OF hIATHEMATICS. UNIVERSITh OF TORONTO. 100 ST. GEORGE ST.. TORONTO. 
ONTARIO, CANADA IISS 3G3 
E-mail address: sml@math.toronto, edu 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Nunlber 7, Pages 2795-2812 
S 0002-9947(03)03276-8 
Article electronicMly published on Nlarch 12, 2003 

LARGE RECTANGULAR SEMIGROUPS 
IN STONE-(ECH COMPACTIFICATIONS 

NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

ABSTRACT. VP show that large rectangular semigroups caI he round in certain 
Stone-Cech compactifications. In particular, there are copies of the 2 « x 2 « 
rectangular semigroup in the smallest idem of (fiN, +), and so, a semigroup 
consisting of idempotents can be embedded in the smallest idem of (fiN, +) 
if and only if it is a subsemigroup of the 2 ¢ x 2 ç rectangular semigroup In 
fact, we show that for an 3" ordinal ,k with cardina.lity at most ¢, fin contains a 
semigroup of idempotents whose rectangular components are ail copies of the 
2 « x 2 ¢ rectangular semigroup and form a decreasing chain indexed by ,k + 1, 
with the minimmn component contained in the smallest ideal of fiN. 
As a fortuitous corollary we obtain the fact that there are <L-chains of 
idempotents of length ¢ in fiN. \Ve show also that there are copies of the 
direct product of the 2 c × 2 c rectangular semigroup with the free group on 2 ç 
generators contained in the smallest idem of fiN. 

1. INTRODUCTION 
The Stone-Çech «ompactification of the integers fin has a semigroup structure 
which extends addition on N a.nd has significant applications in Ramsey Theory 
and topological dynamics. Some questions about the algebra of fiN, which sound 
deceptively simple, have been found to be extremelv ditïtïcult. For example, it 
is hot known whether/3N contains any finite semigroups whose members are hot 
all idempotent. VChether there were two idelnpotents in /31%1 whose sure was an 
idempotent different Kom either remained an open question for several vears. It 
was answered in the aflïrmative in [101, in which it was shown that a certain finite 
rectangular semigroup could be embedded in/31%1. (A semigroup is rectangular if 
and onlv if it is isomorphic to the direct product of a left zero semigroup and a 
right zero semigroup. A rectangular compo'net of a semigroup of idempotent, is a 
nmximal rectangular subsemigroup. As suggested bv the naine, distinct components 
are disjoint. The components are partially ordered bv the relation /9 _< Q if and 
only if/9 0 C /9, equivalently 0/9 Ç /9 [7. Theorem 1].) In this paper, we show 
that the rectangular semigroup 2 « x 2  with the first factor being left zero and 
the second right zero. can be embedded in /31%1. Indeed. pin contains semigroups 
of idempotents which are the union of ¢ rectangulai components each isomorphic 
to 2 « x 2 «. We shall show also that if S is an infinite cancellative semigroup with 
cardinalitv e:, then S \ S contains a semigroup of idempotents which is the UlliOll 

Received by the editors April 12, 2002 and, in revised form, November 14. 2002. 
2000 Mathematscs Sub3ect Classification. Primary 20Ml0; Secondary 22A15, 54H13. 
The first author acknowledges support received from the National Science Foundation (USA) 
via grant DMS-0070593. 

@2003 American lklathematical Societ 3 



2796 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

of at least t rectangular COml)onents, each isomorphic to 2 2" x 2 , with the first 
factor being lefl, zero and the second right zero. 
X first review terminology used in the topological theory of semigroups. Let 
,q' 1)e a semigr¢mp and a topological space. For each s  .', we define mappings 
A. and p. from S to itself l»v A,(/) = .«t and p(t) = ts. ,ç is said to be a right 
lopologicol semigroup if p. is contimous for every s  S. In this case, the topological 
tenter A(S) of S is defined by A(S) = {s  S • A is continu(nlS}. S is said to 
l)e a ,semitot)ological semigrou I) if ils and As m'e hoth contimous %r every s G ,ç. 
It is sai(l te) t)e a topologicol scmigroup if the semigroup Ol)eration is a continuous 
lnal)ping from .%' X ,5' to 
If ,S' is a discrete semigronp, we regard its Stone-'ech «ompa«tification 5' as the 
spaçc of ultrafilters defined on ,5 . with the topology defined bv choosing the sers of 
the fonn 4 = {p G/35 • .4 G 1»} as a baso for the open sers. t3S is then a compact 
tIausdorff simce and . = ('ÇBs(4). XX]' regard S as a subset of 3S, b identifying 
oach clé,mont of .h' with the principal ultrafiltor that it defines. N can he given a 
scmigroup structure which extonds the scmigronp structure of S in snch a way that 
• b' is a compact right topologiçal scmigronp, with ,S" contained in its topological 
('enter. If A ç S..I* will denotc A  A. 
XX3 • shall nso basiç algehraiç 1)r»i)crtios that hold in all compact Hansdorff right 
topologi«al scmigroups. (XX shall ho assmning that ail hypothesized topological 
spaces are ttausdorff.) A simple and important property is that everv compact 
right topological semigroup ç contains an idemt)otent. T has a smallest ideal 
K(T), which is hoth the union of ail the lnillimal left ideals and the union of all 
the nlillinlal right ideals of T. Every right idem of T contains a lnillilnal right ideal. 
and every lefl ideal of T contains a mininlal left ideal. If L is a minimal lefl ideal 
and B a lnininlal right ideal in T. tllen RL = R  L is a group. So B  L contains a 
unique idempotent. If f • T  T' is a honlolnorphisni ff'oin T onto a COlnpact right 
topologi«al grou I) T'. thon f[K(T)] = K(T'). For each minimal right ideal R' of 
T', there is a minimal right ideal R of T for which f[R] = R'. The corresponding 
statement holds for left ideals as well. There are three natural orderings of the 
idempotents of T defined 

C<_Lf <= e = cf , 
e<_Rf  e= fa. and 
c<_f  of=fa=c. 

An idempotent e is minimal with respect to any or ail of these orderings if and onlv 
if e e K(T). The reader is referrcd to [1], [6], or [9] for proofs of these statements. 
When S is a discrete semigroup, the smallest ideal K(/3S) is of special importance 
for combinatorial applications, and in particular, the members of idempotents in 
K(/3S) have strong combinatorial properties. (Sec [6, Chapter 14].) Thus we are 
especially interested in those scmigroups of idempotents that tan be embedded in 
the smallcst ideal of/_S. 
As we have already mentioned, a semigroup S is rectangular provided it is iso- 
morphic to the direct product of a left zero semigroup with a right zero semigroup. 
This is equivalcnt to saying that it satisfies the identities .r 2 = x and .ryz = xz. 
(The necessity is trivial. For the sufficiency, pick x  S, note that Sx is a lefl 
zero scnligroup, xS is a right zero semigroup, and the fimction (a. b) - ab from 
S.r x xS to ,_q is ml isonlorphisnl.) We observe that a rectangular semigroup S 



FIECTANGULA1R. SEMIGIROUPS IN STONE-CECH COMPACTIFICATIONS 2797 

satisfies S = N(S) = LR  L x R. where L deuotes anv minimal lefl idem and R 
any nlinilnal right idem in S. 
If S is a senligroup, E(S) will denote the set of idempotents iu S. 
If A is anv set, 72I(A ) will denote the set of finite nonempty subsets of A. 

'2. TtlE SEMIGIOItPS 
The subsemigronp ]HI =  cg(N2 '') of (tiN, +) hokls ail of the idempotents 
 L 
of ÇN and lllllc of the klloWll algebraic structure. (Sec [6, Section 6.1].) It oecllrs 
widelv in the stndv of semigroups of the fi»rm /3,% If S is an infinite discrete 
cancellative semigroup, every G6 subset of S* that «ontains an idempotent, contains 
copies of N [6, Theoreln 6.32]. N also bas the property that anv compact right 
topological semigroup with countable dense topological eenter is the ilnage of 
uuder a contimmus honomorphism [6. Theorem 6.4]. In this section we introduce a 
senligroup . whi«h satisfies a similar conclusion fin" an arbitrarv infinite cardinal 
. As a consequence of the results of thc next section we shall conchule that ea«h 
N. contains large re«tangular subsemigroups. 
Definition 2.1. Let h be an infinite cardinal. Thon 
supp(z) : {t < n " .ca ¢ 0}. For et < n, eu is that nlember of II such that 
supp(ea) = { a }, and 
The structure of  is that induced bv an "'oid'" as imroduçed bv John Pym [8]. 
When we sav that two structures are "topologically and algebraically isomorphic". 
we nlean tllat there is one fim«tion hetween them that is both an isomorphism and 
a honeomorphism. 
Theorem 2.2. The compact right topological semigroups  attd  are topologi- 
call9 and algebraicall isomorphic. 
Pro@ [6, Theorem 6.15]. 

It is a fact [6. Lemma 6.8] that ail of the idempotents of J3N are in IHI. Thus, bv 
[6, Theorem 1.651, Iç(]HI)= Iç(3N) a IHI. 

Theorem 2.3. Let S be a countablg mfinite discrete gT"oup. Then ris \ S contains 
a topologi«al and algebrai« copy T of IHI such that Iç(T) = Iç(/3S) ç T. 

Pro@ Take any idempotent p C K(/3S). By [6, Theorem 9.13], there is a left invari- 
ant zero-dimensional Hausdorff t.opology on S in which the ultrafilter p converges 
to 1. Then by [6. Theorem 7.24], with X = G = V(a) = S for everv a C G, there 
is a topological and algebraic embedding f : IHI --* ris \ b' such that p  f[IHI]. It 
remains to apply [6, Theorem 1.65]. [] 

A similar result applies to the senligroup (N,-). Given n G w we define the binary 
support of n by n = Etesupp2(n ) 2 t and supp2(O 

Theorem 2.4. Let S = (N, .). There is a topological and algebraic copg T of 
contained in /3S\S which contains all of the idempotents of S\S. In particular. 



2798 NEIL HINDMAN, DONA ST1RAUSS, AND YEVHEN ZELENYUK 

Pro@ Let (pi)ici__ 1 be the sequen«e of primes. Then (Oi-----1 03, --) is isolnorphic to 
(N, .) via the lllftlï) 
let tri : S  co be the projection to the ith factor and let i : /3S -- /3o be its 
contilmous extension. 
Let {Xi : i • N} be a partition of o into infilfite sers and for each i • IN, let 
c)i : Xi -- o be a bijection. We define 0 : co -- S by agreeing that for each i • N 
and each n • 

where ïje 2 e'(j) = 0. k ' note that 0 is a bijection. (If j  supp(n) supp(m), 
then for Solne i, j G Xi and so i(j)  sui,P2 ((n)) supp 2 (i(m)). Also, given 
z  S. fbr each i  N let  = i-l[supp (i(x))], let Z = ie }, and let 
' = tez 2t" Then 0(n) = .r.) Colsequently by [6. Exercise 3.4.1] the continuons 
extension  •   S of 0 is a bijection. SillCe 0(n + m) = 0(n) +0(m) if 
supp2(n )  supl)2(Ttt) = ,  is a holnOlnorphism on  1,y [6, Lelmna 6.3]. 
lb çolnplete the proof, let p be an idempotent iii SS. Since 
a[NI= [N,=ll{0}: N,,=0[]k{0}. 
it suces to show that for all n  N, 0[2"]  p. So let n  N and suppose that 
0[w2 ' ]  p. Pick t e {1,2 ..... 2'- 1} su«h that 0[w2  + t] e p. pick j e supp2(t), 
and pick i such that j  Xi. Now Yi(P) is an idempotent: so either (p) = 0 
or (p)  flNN. Tlms by [6, Lelnlna 6.6] w2 e'(j)+l  p. Pick z  w2 ()+1 N 
i[O[2'+t]] and pi«k k e 2"+t such that .r= i(O(k)). Then j e supp2(k)NXi; 
so i(j)  stt)P2(x), contradicting the fact that z  2 (j)+l. 
Observe that if  > w, then N N K(ll ) = . To see this, one lets p  N and 
q e Nk<w d {e IlÇ{} • nlill sllpp(w) 2  and lllax supp(z) < w'}. 
Then p  I1  + q + p and so [6, Theoreln 4.39] applies. 
Theorem 2.5. Let  be an inflnite cardinal and let T be a compact right topological 
semigro,p. Assume that there is a set A ç A(T) s,ch that lAI   and A is dense 
in T. Then there is a contim, o,s s,wjective homomorphism f " N  T. 
Pro@ Elmlnerale A as {t : a < }. with repetition if lAI < . Let {I :  < } 
be a partition of  into subsets of size . Define h • IV + T by first agreeing 

that for each o. < t;, h(e,) = tx, where o. • I. 
h(_.a F ea) = HaF h(ea), where the product is 
indices. Define h(0) arbitrarily. Let ï :/3II, -- T be 
and let f be the restriction of  to IE,. 

Then for F • 7)(t), define 
taken iii increasing order of 
the contimous extension of h 

To see that h [IE,] = T, it suffices to show that A C_ h[lE,]. Given "7 < t, we have 
that II[ = t. Pick a t-uniform ultrafilter p on {ec • a • I-}. Then p • IE, and 
h(p) = tv because f is constantly equal to t- on {ec • a • L}. 
To see that f is a homolnorphisln it suffices by [6, Theorem 4.21] to observe that 
whenever a: • IIÇ\{} and / • IlÇ\{} with rein supp(/) > max supp(x), then 
(z + v) = h(z). (v). [] 

Definition 2.6. Let S be a selnigroup, let h; be a cardinal, and let (t,x),x< be a 
-sequence iii S. 
(a) Given F • 7)l(n ). 1-leF t is the product in increasing order of indices. 



RECTANGULAR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2799 

(b) If D C_ h;, then FP((t)aD) = {IIF ta • F E PI(D)}. 
(c) The sequcnce (ta)a< has distinct finite prodacts if and onlv if whenever 
N. G E PI(A) and 1-[,XeF t,x = 1-[aec fa, one must bave F = G. 

Theorem 2.7. Let S be an infinite cacdlative discrete se'migroup with «ardmality 
ris \ S coutoins a topological ad algebraic copy of ,. 

Pro@ By [6, Lemma 6.31], we nlay choose a t-sequen('e (ta)a<,, in S with distinct 
finite products. Let T = FP((t)x<). For each V < , let Tv = FP((t),<x<). 
U put ç = < «s(T). By [6. Theorem 4.20],  is a subsemigroup of riS. 
 define 0 • T  I1 by 0(xe  tx) = xe «» (Since the sequence (t)a< 
has distinct finite products, the fimction 0 is well defined.) Let  clos(T)  flllç 
denote the continuous extension of 0. By [6, Theorem 4.21], the restriction of g to 
ç is a homomorphism. Nov  is injective, by [6. Eercise 3.4.1]. Since for each 
7 < , [T] = cft3u-{x e II/ • lnil SUl)p(x) > 7},  maps  onto N. Thus  
determines ail isomori)hisnl from  OllIO .  

3. CHAINS OF RE('TANGI'LAR SEMI(IROI'PS 1N - 
Let  be an infinite cardinal and let IÇ denote the rcctangular semigroup 
with the first factor 1)eing left zero and the second right zero. We show in this 
section (in Crollary 3.10) that for mly infinite cardinal h;. algebraic copies of 
can be round in tç(NI,). Indeed. if A is any ordinal for which I)q _< , there is 
a (lecreasing chain (D»)»<a of diioint copies of I.), contained in NI, with Da 
embedded in tç(NI,). 
Notice that 1/ contains a copv of cvery l'ectangular semigroup of cardinalitv at 
lllOSt 

Definition 3.1. Let A be any ordinal and let A be all.V llOllelllpt.v set. Let 0 denote 
a selected elelllent of A. For p < A. let Cp = A x 4 x {p} and let 
C = Ca,a = A U Up<aCp = A U (.4 x A x A). 
The operation • on C is defined as follows. Let a, b. c, d E .4 and let p. q < A. Then 

a.b = b, 
o. (b,c,p) = (b.c,p), 
(b,c,p).a = (b,a.p), 
(a,b,p).(c,d,q) = (a,d, pVq), 
where p V q is the maximum of p and q. 

V'e leave to the reader the routine verification that the operation on CA.a is 
associative. Notice that for any p < A, Cp is a cop.v of l/iA I. 
Definition 3.2. Let A be an ordinal and let p < A.  let vp = (0,0.p), and 
for every x = (a, b.p) e C, we let xt = (a. 0.0) and Xr = b. For x  A, we let 
The folloxving is silnple, and its proof is like that of [6, Theol'en 1.46]. 
Lemma 3.3. Let S be a semigroup, let H be an ideal of S. let L be a minimal 
l@ ideal of H. let  be a minimal right ideal of H, and let z ŒE S. Then Lx is a 
minimal l@ ideal of H. .rR is a miimal right ideal of H. xL ç L. and Rx ç . 



2800 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

Pro@ It suffices to establish the assertions about La" and 3"L. Now L.r C H.r C H 
and HLx C L.r: so Lx is a left kleal of H. Let M be a lcft kleal of H with M ç Lx. 
Let.l={gL'gxM}. GivcngJandzH, wehavezyLand zyx3l: 
so zg Ç J. Thus .] = L and so M = L.«. 
Next. givcn g G L = Hg, so pick z Ç H such that y= zy. Then xy=xzy Ç 
H9 = L.  

Lemma 3.4. Let A bc o nonempty set with di.çling«ished element O. and let C = 
C.4,1. Let T bc a right lopological semigroup, and let f • T  C be a surjective 
homomorphism for u,hich f-[A] and f-l[Co] are conpact. Then there is a homo- 
morphism g C  T sach that f o g is the idcnlity on C and 9[Co] ç Iç(f-[Co]). 
(f T is compoct, then 9[Co] ç K(T). 

Ptv@ V\, first (lvfine g on .4. We have that f-1 [.4] i8 a COlnpa('t semigroup. Choose 
a minimal right idem Nof f-[A]. For ea«h o  A. f-il{c,}] i . ,,f id(, of f-l[A]. 
So (.hoose a llininal l(,ft id«al « of f-l[A] with S ç f-[{}], ald let g(a) be the 
identity of the gloup N  a- Then immcdiately f(g(a)) = a. Also. given a, b Ç A 
we have that .q(a) and g(b) are idt'mpotents in N: so g(a)g(b) = g(b) = g(ab). 
Let = {(,.o.o):o  4}. Tll('ll  is a l('ft idçal of Ç0: so f-l[B isa left ideal 
,,f f-a[c0] whi«h thelef(n'e contains a minilnal left ideal L of f-l[C0]. For each 
a G A let F = {(o. b, 0) : b G A}. Then F is a right ideal of C0. So pick a lnininml 
right ideal R of f-[C0] with R ç f-lirai. Bv LCIlIIlI& 3.3, SillC f-l[C0] is ail 
i(leal of f-l[A uÇ0], we bave that 9(0)- R is a nfiniliml right ideal of/-l[C0] and 
L. 9(a) is a nlinilnal lcft ideal of f-z[C0]. For a. b G .4. let 9(a, b. 0) be the identity 
of the group 9(0)" R. L.9(b). Notice that if T is COlnpact. then K(T) ç f-[C0] so 
that Iç(f-l[C0])  Iç(T) and thus 9(a, b, O)  K(r). Also 9(a, b. O) = 9(O).x.-g(b) 
for SOllle .l'e a alld SOllle  e L. So f(.q(a,b.O)) = O. f(x). f(y).b= (a.b, 0). 
To conclu(le the proof we nee(t to show that g is a hOlnOlnorphisln. First we let 
o. b, c Ç A and show that g(o) .g(b. c. O) = g(b. c. 0) and g(b. c. O) .a = g(b. a. 0). Pick 
.r  H6 • L such that g(b. c, O) = 9(0) • x. g(c). Then 

g(o).g(b,«.O) 

= (,)-9(0)-.,-9() 
= 9(0)- .- 
= 9(b, «, 0). 

So the first clainl holds directlv. Multil)lying on the left by g(b.c. 0) and on the 
right by g(a) Olm sees that g(b. c. 0)-g(a) is idelnl)otent. Since g(b.c. 0). g(a)  
9(0)" Rb" L. 9(c). g(a) = 9(0). Rb" L. 9(a), we nmst bave that g(b. c, 0). g(a) is the 
identity of g(0) • Rb" L. 9(a), nalnely 9(b, a, 0). 
Finally, let a, b. c, d Ç 4. Then 9(a. b. O) • 9(c, d, O) Ç 9(0) • R, • L • g(b) • 9(0) • 
R«.L.g(d) c 9(O)-R«-L'g(d). So it suffices to show that g(a.b.O).g(c,d,O) 
is idempotent. These elements satisfy 9(a.O.O)  L • g(O) and 9(c, 0,0)  L • 
9(0). So as idelnpotents in the saine lninimal left ideal of f-[Co], we bave that 
g(a, O, O) • 9(c, 0.0) = 9(a. 0.0). Recall that we bave shown that for any x. y  A. 
9(0)-g(x,y,O) = g(x,y,O) and g(.T.y,O)-g(O) = g(x,O.O). Thus we have 

g(a, b, 0). g(c. d. 0). g(a, b. O) 

= 9(a,b.O).g(O)-g(c,d,O).g(O).g(a.b.O) 
= g(a,O.O).g(c.O.O).g(o,b.O) 
= g(o,O.O).g(a,b,O) 
= g(o,b.O).g(O).g(a.b.O) 
= 9(a,b.O). g(a.b,O) = g(a,b, 0). 



IECTANGULAIR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2801 

Multiplying on the right by g(c, d, 0) we have that g(tt, b. 0).g(c, d, 0) is idempotcnt. 

We now cousider the situatiou in whi«h k > 1. For k > c0 we do uot necessarily 
get that g is a homomorphism, hut xve corne 

Theorem 3.5. Let A be a nonenq#.q set u,ith distinguished element O. let A be an 
ordinal, and let C = Ç4.. Let T be a right topological semigmu p, and f : T 
be a su«jectiue t, omomo,vli.m .sud, tlmt f-[A] i.s. compact and f-l[c] is comp,wt 
for every p < A. The'n there is a function g : C  T su«h th«.t f o g is the identity 
attd 9 bas the .followin 9 pvperties: 
(i) If q  p < .  G Cp, and g G AUC. then 9(«g) = g(a')'9(g) and g(g).g(.r) 
is art idempotent in thc saint minimal lefl id«al 4 f-'[G] C,,s 9(.qx). 
(ii) lf p < A. « G C v. and U G 4  Co, then 9(g) " 9(.v) = 9(g.r). 
(iii) ff q  p < A. n G . p = q+n. g G C. and x 
(iv) gp < A. ther, 9Cp]  lç(f-l[cv]). 
The seigrou p T «ontins  semigrou p D = p«A Dp of idempotents where ea«h 
is a ,ecta,gthtr component of D with g[Cp] Ç Dp and the scquence {Dp)p< is de- 
cre«sing in the orderin 9 of components, so that for each p < 
6lAI  IX]  w. then for et, ch p < X. Dp] i.ç i.çomo,phi« fo I]A I. 
Proof For p < X we &'fine g on A U qp Cq 10 induction on p so that g satisfies 
conclusions (i), (ii), (iii), and (iv). By Lemma 3. we mav definc 9 on A U C0 so 
that g satisfies (iv) and is a homomorphism and therefiwe satisfies (i). (il), and (iii). 
Now let (l < p < X and assume that 9 has been defined on 4 U q<p Cq. 
IO show fiist that we moEv choose a minimal left idem L of f-[Cp] such that 
L ç q<vf-l[Cp].g(%)and fiL] = Cp-%. A si,uple computation establishes that 

x. 9(q) 

= .. g(), g(,,,.) 
 f-'[%]. (,,). 

by (i) 

Consequently. r-]q<p f-'[Cpl- g(uq) is a lefe ideal of f-l[Cp] mM thus contains a 
mi,,i,nal left idem L of 
Now giveu .r  L, one bas x  f-a[G] g(uo). So for sou,e «,.  A. f(x) = 
(a,O,p)  G- up. Thus f[L] 
If p is a successor ordinal, observe that g(%_).f-[Cv] is a right id « f-[Gl 
and pick a minimal right ida 
is a minimal right ida « G ad f[l ç %-" G: o f[l = -1" G- 
If p is a limit ordinal, note that f-[%- Cp] is a right idem of f-[cp]. So pick 
a minimal right ideal 
Now f-[cp] is an idem of f-[AuCUCp]. So by Lemma 3.3. for any x  Cp, 
g(xt) - R is a minimal right idem of f- [%] and L-g(r) is a minimal lefl idem of 
f-l[c A. çhr«, (Ot)" 
g(a') be the identitv of 0(.re) • R- L- O(a>). Notice that (iv) is satisfied. 



2802 NEIL H[NDMAN, DONA ST1RAUSS. AND  EVHEN ZELENYUK 

To verify (i), let q < p. let x  Cp, and let y  A tO Cq. To see that g(xy) = g(x). 
g(y}, we show that g(x} • g(g) is an idempotent in the saine grouI) as g(xy). Since 
(J'g)e = xe and by Lemma 3.3, R. g(g) Ç R. we have that g(xg)  g((xge)) " R = 
(-«) • n ,« g(,). ()  (e) • n-() ç (e) • n. 
also. (a.g},. = y and so g(a'g)  L.g((xy}) = L.g(y). ço see that g(x).g(y) e 
L. g(y), we consider two cases. If g e Q, then g(x) • g(g) e g(x) • L. g(Yr) ç 
L" g(y,.). Now assmne that q < p (and y  A U Cq). Note that L ç f-[Cp]. 
g(«q); so g(uq) is a right identity for L and thus L : L. g(uq). Also. a simple 
computation establishes that uuy : uqy. Therefore, using the fact that (i) holds 
at q, g(uq) . g(x) . g(y) : g(«qx) . g(y) = g(t, qxy) = g(uqy) : g(uq) . g(y) and 
th, g().9()  r.9().9(y)= r-9(,)-9(«)-9(Y)= r'9(,)9(>)= 'g(>)- 
(.onsequ¢ ntlv, we have in anv event that g(xy) and g(x). g(g) are members of the 
group g(xt) • R . L. g(y).  show that they are equal bv showing that g(x) • g(y) 
is idempotent. 
Since g(x), g(0) : g(.r,.0) = g({}) : g(g0) = g(y) • g(O) we have that g(x) - 
g(O)  L. g(x) • g(0) = L-g(0) and be«ause g(x) • g(y)  L. g(y) we bave that 
g(«). g(). 9(0)  . g>). 9(0) = L. g(0). 
w g() = 9(.,c). -9() o- ,, z  -n..o 9(0)- 9(*e) = g(0e) = 9(xe) 
and so g(0) • g(x) : g(xe) . z. g(x) : g(x). If y G Cpwe bave similarly that 
g(O). g(y) : g(y), while otherwise g(0)- g(y) : g(Oy) : g(y) by (ii) of the induction 
hypothesis.  have that g(x)-g(0)-g(x), g(O) : g(x) . g(x) . g(O) = g(x) . g(O). 
So g(x)-g(0) is an idempotent in L. g(O), which is a minimal left ideal of f-l[cp 
by Lenmm 3.3. Therefore g(x) • g(0) is a right identity for L • g(0) and thus g(x) • 
9(). g(0). g(z). g(0) = 9(-). 9(). 9(0). So 

g(x). g(y). g(x). (y) 

= v(,). v(y). g(0). v(,), g(0). 
= g(x). v(). v(0). v(v) 
= (,). g(y)-(y) = («). 

as required. 
Bv Lemma a.a. oy).g(xe). R ç f-l[cp]; so (y)'g(x) E 
L-g(«,.). Also, g(yx) e L. g((gx),) = L. g(x,) and by Lemma a.a. 
is a minimal left ideal of f-l[Q,]. To see that g(y) • g(x) is idempotent, note 
that xyx = x. So g(x). g(y). g(x) = g(xg)- g(x) = g(.'gx) = g(x) and t5us 
g(g) . g(x) . g(y) . g(x} = g(g) . g(x), as required. This completes the verification of 
(i). 
Toverify (il), let. x C, and let y  AtoC0. Pick z L-Rsuch that g{x) = 
g(xe) • z-g(x,). Then g(y) • g(x«) = g(yxe). If y  -1, then yxe = xe so that 
g(y).g(x) = g(y).g(xg).z.g(x,) = g(xg).z-g(x,) = g(:r} = g(yx). So assume that 
g E C0. Then yxe = (yx)e and (y«), = x,. So g(y). g(x) = g(y). g(xe)" z. g(x,,) = 
.q(yxe). z.g(x,) --- g((yx)t), z.g( (yx),). So to see tiret g(y).g(x) = g(yx) it suffices 
to recall ïrom (i) that g(y) • g(x) is idempotent. 
To veriïv (iii), let n  w and let q <_ p such that p = q + n. Let x  Cp and let 
g ff Cv. If n = 0, the conclusion ïollows ïrom (i). So assume that t > 0 so that p is a 
suc«essor ordinal and p- 1 >_ q. Now (ya')e = ye: so g(yx) ff g((yx)g). R = g(gê).R. 



IRECTANGULAR SEMIGIROUPS IN STONE-CECH COMPACTIFICATIONS 2803 

Recall/hat R C_ 9(up-) • f-x[C v] and conse<lUently R = 9(up-,)" . Thus, 
9(9)-9()  9()9(,r«)-  
= g()'g(,r«)'g(llp_l)" 
= g(9.rc)-g(v-)" R by (i) at q 
= g(,rftt, p_l). R by (iii) a.t p- l 
= 9(yeup-)" R 
= g(Ye) "g(up-1) • R by (ii) at p- 1 
= g(e) . 
Now 9(Y)" 9(x) is an idempotcnt in thc sa.me minimal left ideal of f-[Cp] by (i). 
Since g(yx) and 9(Y) " g(x) arc also in thc saine minimal right ideal g(y() • t. they 
must be equal. This complotes the induction step. 
Next, we establish (v). So assume that T is compact and A is a successor. Thon 
Iç(T) Ç f-l[C_l] and so h-(f-l'[C_]) ç h'(T). Sin«e g(x) e lÇ(f-l[CA_l]) for 
every x  C-l, (v) holds. 
5r each p < A, let 
 = { (') • F e ç«(a), » = ,,,x F, ,,a fr «h q e F, ' e C}, 
where for each F, thç product qF g(,rq) is taken in increasing order of indices. 
 show now bv induction on IF] that 
ifp < A. y  Cp. F  çy(A), xq  Cq for each q  F, 
(*) and maxF  p. tlwn g()'qF g(q) = g(Y'qF q)" 
Let r = maxF. If F = {r}, then g(y)-g(x) = g(yx) by (i). So assume that 
() « (.) = () a (Æ)a() 
= 9(Y" ea xq) • g(x.r) by the induçtiou hypothesis 
= a(a r) b (i). 
Now we show that each member of D is idempotent. So let F  çy(A). let 
p = nmx F, and for each q  F, let xq  Cq. If F = {p}. then g(Xp) is idempotent. 
So assume that IFI > 1 and let a = F  {p}. Then 
« a() « () = a a(«) a(«) « () 
= a («) 
= a ()a( 
= a ()a() 
= qF g(.rq) . 
Now let r.p < A, let a  Dr, and let b 
have that a = HqeF g(xq) and b = Hqea g(Yq) where maxF = r. maxG = p. each 
xq e Cq and each yq e Cq. If r < llin a, then ab = qeF g(zq) . qea g(Yq) e Dp. 
Ifr k p, thenab = qeFX{-r} g(xq)'g(xr'qG yq)e Dr (where the qeFX{} g(xq) 
term is simply omitted if F = {r}). So assume that rein G 
G " q  r}, and let L = {q e G • g > r}. Then ab = HqeF{} g(xq).g(x. 
,u).  (u) e D. 
Thus D is a semigroup of idempotents and for each p < A, K(Uq p Dq)  Dp. 
Let r  p, let o = qeF g(xq) e Dp, let b = qG g(Yq)  Dr, and let c = 



2804 NEIL HINDMAN, DONA STRAUSS, AND YE¥ HEN ZELENYUK 

H,eH g(zq) • Dp. Then 
abc = H 
qF\{p} 
= II 
qF\{p} 
qeF\{p} 

qG qH 
(.1. (» . H .q " H z) 
qG qH 

g(.rq)- g(XpZp) 

alld 

(c - 

H g(.rq)-g(.rp- H 
qF\{p} qH 
qeF\{p} 

So abc = ac and so each Dp is a rectangular subsemigroup of D. To see that D v 
is a rectangular component of D, suppose that a • Dp and b • Dq, where q < p. 
Then f(bob) • C and f(b) • Cq, and so b,b ¢ b. To sho« that Dp  IiA I if 
lAI > [,Xl > a, we observe that Cp contains a left ideal L and a right ideal R, each 
with 1,4[ elements. If a,b • L, then ab = a and so g(a)g(b) = g(a). Thus g[L] 
is contained in the left ideal Dpg(b) of Dp. Silnilarlv. g[R] is contained in a right 
ideal of Dp. So Dp contains a left ideal and a right idem each with at least 
elements. They cannot have more than lAI elements because for each F • 
with p = lnaxF, there are lAI IFI = lAI choices for 1-[qeF g(Xq). So [Dp[ = 141. 
Thus D v , L x R . IAI- [] 

Two ol»vious questions are raised by Lelnlna 3.4 and Theoreln 3.5. First, can 
the flmction 9 constructed there be required to be continuous? Second. tan the 
flmction 9 in Theoreln 3.5 be required to be a honollorphism? We shall answer 
bot h of these questions in t he negative, even w hen the stronger requirements that T 
and (7 be compact and (7 be a topological semigroup are added. We shall ha:ce lmed 
of the following lemma, whose routine proof we omit. (Recall that anv successor 
ordinal is a COlnpaçt Hausdorff space under its order topology.) 

Lemma 3.6. Let .4 be a compact space, let A be a ordinal, let A x A x (A+ 1) bave 
the product topology, avd let A and A x A x (A + 1) be clopen subsets of C = CA.X. 
Thev (: is a compact topological semigroup and (' = K(C). 

We now show that. eVell for A = 0, one cannot require that g be continuous. 
"Ve remind the reader that an F-space is a completely regular space A" in which 
{x • X : f(x) > 0} and {x • X : f(x) < 0} are completely separated for ail 
continuous f : X  

Theoreln 3.7. There eïist a nonempty set A. a topology o'n C = CA, 1 8uch that 
C is a compact topological semigroup and A and Co are compact subsets of C. a 
compact right topological semigwup T. and a continuous surjective homomorphism 
f : T -- C such that there is no cottinuous homomorphism g : 6' --} T for which 
f o gis the identity on C. (In fact. there is no cotinu.ous injective function from 
C roT.) 



RECTANGULAR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2805 

Pro@ Let A = pIN, let C = fin U (fin x fin x {0}) with the topology given in 
Lelnlna 3.6, and let T = H. Then N U (N x N x {0}) is dense in C = A(C). So 
there is a continuous surjective homolnorphism f • T  Ç by Theorem 2.5. 
Now suppose there is a continuous injective flm«tion 9 "   T. Then by 
Theorem 2.2 there is a continuous injective flmction ff'oin Ç to  ç fiN. But this is 
ilnpossible because fin is an F-space [3, Theorem 14.25]. So evel'v compact subset 
X of/IN is an F-space, because everv COlltillUOllS function fronl .ç to [0,1] ll a 
continuous extension to fiN, by the Tietze extension th(,oroln. But fin x fin is llOt 
an F-space by [3. 14Q]. 
Theorenl 3.8. There exisl a nonempt set A u,ith distinquished el«ment 11, a topol- 
o9 on C = CA,+I suc tat C is a compact topological semigrou p aud A 
are compact su.bsets of Ç for each p  ', a compact rtgt topological semigroup 
T, and a continuous surjective homomorpisnt f " T  C suc tot lhere is no 
homomotph.ism 9 " C  T for whi«h f o 9 is the identit.q o ('. 
Pro@ Let A = {0} raid let C = CA.+a with the tolmlogy given in Lemma 3.6. Let 
u0 = 0. forp < w, let up+a = (O.O.p), and let u = (0.0. w). Th(,n C = {p "p N w} 
811d Upq = llpv q fi)l" ail p, q  w. "[blologically, u is the onlv non-isolated point 
in C Let (vp)v<  be a sequence of distinct points none of which are in C. Let 
T = {u v • p < ,} U {vp • p < «} and for p.q < ' define an operation on T as 
follows: 
IIpq 
IIp'q 
l'pq  Up'q  p . 
b leave it to the reader to verifv that the operation is associative. 
Let ç {e0} be discrete and let T be the one point compactification of T {v0} 
(with t'0  the point at infinity).  claire that T is a riht topological semigroup. 
Let p < . To ste that p is contimous at v0. let Il be a lmighborhood of 
'0 = (0) .d t U = U'({" 
To see that p, is contimous at v0. let II be a neighborhood of v0 = P,(0) and 
t u = {% . p  q <  ..d  
Define f" T Ç by f(up) = up and f(vp) =u. for eachp < w. Then f is a 
continuous surjective honlomorphisln. Suppose that g • C ç is a honlomorphisnl 
for which fog is the identity. Then for p < 
such that g(u) = vq. But then. vq+ = 
g(u) = eq. a contradiction. 
 shall see next that we can get the function 9 to be a honomorphism bv 
requiring that T be Selnitopological. This corollary tan then be viewed as saying 
that Ç is something like an absolute co-retract in the category of semitopological 
semigroups. Ç becomes an absolute co-retract in the category of compact semi- 
topological senfigroups if it is given a topology for which it is in this category with 
A and each Çp being compact. 
Corollary 3.9. Let A be a nonempty set wilh distingished elemenl O. let A be an 
ordinal, and let Ç = Ç.4.a. Let T be a senitopological semigroup, and f • T  Cbe 
a continuous homomorphism such that f-'[A] is compact and f-[Cp] is compact 
for every p < A. Then ther« is a homomorphism g ' C T such that f o g is the 
identity. 



2806 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

Proof. At stage p of the induction in the proof of Theorem 3.5 one has that for 
each q < p, g(Uv)- f-l[cp] is a compact right ideal of f-[c]. So one may choose a 
minimal right ideal R of f-l[cp] with R Ç q< g(uq), f-[%] and f[R] 
Then. if  G Cq for some q G p and G Cv, just s one showed in the verification of 
hypothesis (i) that 9(x) • 9(P)  L- 9(g), one can show that 9(9) "9(w)  9(9t) " R, 
so that 9() " 9(x) and 9(9x) are idempotents in the saine group. If 9  A and 
x  Cp, then 9(U)" 9(x) = 9(u.r) by (ii). ( did hot need to consider the case 
y G A separately at that point in the proof of Theorem 3.5 because the equation 
qr = tlqr '-q valid in any event. The corresponding equation y.reUq = ç(Uq is 
hot valid if y  A.) 
 now prescrit some immediate consequences of Theorem 3.5, although with a 
bit more effort, wc shall get a stronger result, namely Theorem 3.16. 
Corollary 3.10. Let  be an ifinite cardinal and let A be an ordinal with 
Then  cotains a subsenigroup D = UpA Dp of idenpotents wher¢ each Dp 
a rectangular comportent of D isomorphic fo I ad the seqence (Dp)A is de- 
çasing i the ord('ring of çompoent. o that for ea«h p  A. Dp = K(UqS  Dq). 
Proof. Let a have the discrete topology and let 4 = fla-. Let C= 4.+ and let 
lmve the topology describcd in Lemma 3.6. Let T = . Since U( x  x (A+ 1)) 
is a dense sui»set of C = A(C), by Theorem 2.5 there is a continuous surjective 
homomorphism f • T  C and so Theorem 3.5 applies. 
Corollary 3.11. Let ,S' be n infinite cancellative discr¢te semigroup with cardi- 
ality  ad let A be an ordi,al with lAI G a. Then ils  S contabs a subsemigrop 
D = UpA Dp of id¢mpotents where ¢ch D is a r¢ctngdar coponent of D 
isomovphic fo l/ and the seqence (Dp)pA is decreasing i the ord¢ring of com- 
o¢t., o t.t fo ¢« p  . Dç = ç(U; G)- If s = (N, +), s = (N,-), o 
S is  coudabl nfinite discrete gmp. then D ç K(flS). 
Proof. By Theorem 2.7, ils  S contains a topological and Mgebraic copy T of . 
(If S = (N, +), choose T = . If S = (N,-) or S is a countably infinite discrete 
group, choose T  in Theorem 2.4 or Theorem 2.3 respectively.) Then Corollary 
3.10 applies. 
If S = (N,+), S = (N,.), or S is a countably infinite discrete group, then 
K(T) = K(çS) aT So bv Theorem 3.5, with A+ 1 in place of A. we bave 
[c] ç K(ç)  ç(S) ana so D ç K(S). 
Corollary 3.12. Let S be a contabl infinite discute grop. Then there is a copy 
of ç« contained in K(flS). 
K can completely clmracterise the semigroups of idempotents that can be cm- 

bedded in K(flN). 
Corollary 3.13. 
nality t and let D 
(i) There is a 
(ii) There is a 
Proof. Conclusion 
3.11. Assulne now 
[D I _< I/3NI = 2'. 

Let S be an infinite cancellative discrete semigroup with cardi- 
be a semigroup of idempotents. 
copy of D in ris \ 5? if D is rectangular and lOI < 2 2" . 
copp of D in K(flN) if and only if D is vectangular and [D[ < 2'. 
(i) and the sufficiency of (ii) follow immediatelv from Corollary 
that D is a semigroup of idempotents contained in K(flN). Then 
Next observe that any subsemigroup of idempotents in K(flS) 



IRECTANGULAIR SEMIG1ROUPS IN STONE-ECH COMPACTIFICATIONS 2807 

must be rectangular. To sec this, sui)pose that a', y, z E K(13ç). Then zz and r9z 
belong to t.he saine minilnal left ideal and t.o t.he saine nlinimal right, ideal. Hence, 
if they are i(telnpotent, they lnUSt, be equal. 
Recall that any two nlaXilnal groups in the slna.llest ideal of a COlnpact right 
topological semigroup are isomorI)hic. çç sec that we can get the direct product of 
such groups with an embcdded rectangular Selnigroup iii tlle slnallest ideal as well. 
Theorem 3.14. Let T be a compact right topological sew, igroq», let D be a rect- 
a9u.lar su.bsemigroup of K(T), ad let G be a vm'inal subgrou p of K(T). There 
is an algebr«ic copy of D x G covtaived "i K(T). 
Proof. Let L be a lnilfimal left i(leal of D and let R be a lninimal right ideal of 
D. Since D is reçtangula.r, D is the internal direct product of L and R, lnealfing 
that each element x of D can be written mliquely as x = xx where x G L 
and xn G R. Also, RL = R  L is a subgroup of D and so, sinçe D consists of 
idelnpotents, RL = {e} for SOlne e. Then for any x,y  D, ,rnyc = e. Note a.lso 
that (xy) = xc and (.ry)n = y. 
e lnax" assume th«tt G = etc. Dofine  : D x G  K(T) by (x, g) = 'Lg,rR. 
 claire that  is an injective honlomorphism. Let (, 9), (Y, h) 6 D x G. Then 
ç((x, g)(y, h)) = xLg.rnyz hyn = .r Lgehyn = (,ry)Lgh(.ry)R = ç(,ry, gh ). 
Now assume that (x, g) = 9(Y, h). Then g = ege = ,rRxrg.rR.rL = .rnyrhyR.rL 
= che = h. Also, xLT  yLT ¢ 0 and xLT and yLT are minimal right ideals of 
T; so xLT = yrT. Similarly T.rn = Tyn. Now .r = 'L,rR G .rrT  Tan and 
  yLT  TyR. So ,ç and y are idempotents in the saine gronp and therefore 
,r= y. 
Corollary 3.15. K(N) cotains an algebraic copy of 1, x F, whe I5, is the 
2 « x 2 « rectangldar .semigroltp ad F is the free group on 2  genemtors. 
Pmof. Bv Corollary 3.13, K(N) contains a copy of the 2  x 2  rectangular Sllli- 
group, and by [4], each maxinlal group in K(N) contains a copy of the free group 
on 2  generators. Therefore the result %llows Kom Theorem 3.14. 
We now present, a strengthening of Corollarv 3.10, producing a longer chain of 
rectangular components. ecall tha/ the Sosliz zumber S(X) of a topological 
space X (also known as the cellalarity of X) is the least cardinal 2 such that. X 
do no/ bave a collection of 7 pairwise disjoint nonempty open subsets. Sec [2, 
Chapter 12] for considerable information about the Souslin number of t.he space 
U(n) of uniform ultrafilters on -. ecall. in particular, t.hat the Souslin number of 
N* = çNN = U(N) is 
Theorem 3.16. Let  be an infinite cardinal and let  be an infinite ordinal for 
c I[ < S(U()). There ezist a set A with lAI = 2 " ,,d , to g 
if p = q+n for some  < w, then 9(g)'9(x) = 9(gx). Also,  contains a semigroup 
D = p< Op of idempotents whe for each p < , Op is a rectagular comportent 
of D isomorphic to ç, 9[Cp] ç Dp, and the sequence {Dp)p< is decasin 9 in 
s.ccessor, then D_ ç K(). 
Proof. Since lAI < S(U(n)), choose a fanfily {Ep)p< of subsets of  such that each 
[Ep[ = n and Ep  EqI < ¢ when p ¢ q. For each p < A we define 



2808 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

by bp(w) = Yeoepns,pp(,) e (where e0 e = 0) and let 
be the continuous extension of çp. If v, u,  11 and supp(v)  supp(w= 0. then 
0p(v + u,) = çp(v) + 0p(w). So by [6. Theoreln 4.21] the restriction of p to N is 
a hollloHorphisllL 
Next observe that for x  N, p(z)  {0}UN. Ifthere exist B  z and 
o < n such that supp(u,) Ep = 0 whenever w ¢ B and rein supp(u,) 
then ç(x) = 0 because çp is constantly 0 on {u' ¢ B • rein supp(u,) 
Otherwise {0p[{« e B • rein supt)(w) OE }1 •  e - .d  < } has the finite 
intersection property and so is contained in ail ultrafilter . This   N and 
Let T0 = N  c?{w  II • supI)(U,) ç E0}. Notice that T0 is a compact 
subsemigroup of N. For each p with 0 < p < A let 
T o= {x Ç N • çp(x) e N and for all q with p < q  A, çq(w) = 0}. 
To sec that T v ¢ , let .r be a mfiform ultrafilter on {e • a  Er. If q ¢ p, then 
[Eq  Er[ < . So çq(.c) = 0. while çp(x) = z  N (because çp is the identity on 
{« • a:  Ep}). Since q is a homomorl)hisln on N for each q  A we has'e that 
Tp i8 a s/l])Sellligrollp of . Sillce Tp :   -1[]  p<qA 
compact. 
If A is a successor, let T_ = N  v v-[N]. Then T_ is clearly 
compact subsemigroup of N provided T_ ¢ 0. XX show in fact that 
T_. Let x  K(N), let p < A. and let 9 be a uniforln ultrafilter on {e • 
a G Ep}. By [6, Theorem 4.39] pick z G N such that m = z+9+w. Then 
0,«) = ç,:) + ç,u) + ,) = ,) + v + ,) ¢ 0. 
Next observe that for p, q < A, T v+ Tq = Tpvq and if p ¢ q, then T 
Let T = p< Tp. If T has the relative topology induced bv N, T is a right 
topological senligroup. 
Let A = N  cg {e "   E0}. Then A is exactly the set of uniform ultrafilters 
on {e "a  E0}, and so IA : 2 2. Let C = Ca.. 
X% shall now construct a smjective homomorphisn f • T + C XX first intro- 
duce some mappings. Let 0 • II  {e • a  0} be a flmction whose restriction 
to {e • a  E0} is the identity, whose restriction to {e • a Ç E} is a bijection. 
and whose restriction to II  {e • a  E0 U E} is a bijection. (In particular. 0 is 
at most three-to-one.) Let 0 • 1I  d {e • a  E0} be the continuous extension 
of 0. 
Let e(0) = a(0) = 0. For w e I1{0}, let e(w) = e where 7 = max supp(w). If 
supp(w) ç E0, let a(u') = 0. Otherwise let a(,) = e where o = min(supp(w)E0). 
t -  + {0}  d{« - < «} ,,a - d  {0} u d 
the continuous ext.ensions of a and e respectively. Notice that a is the identity on 
{e " O e E0} alld e is the identity on {e " a < }. So ? is the identitv on 
N  d {e • a Ç hE0} and 7 is the identity on N  cg {e • a < -}. X claire 
that for «.  G 

-¢(x + y) = è(y), 

(,) 

(.r+B) =  (x) ifxTo, 
(y} if.rTo. 



IRECTANGULA1R SEIG1ROUPS IN STONE-('ECH COMPACTIFICATIONS 2809 

For ¢v.v E II,, if max supl,(V) > max supp(w), then ((w + v) = e(v) o that 
è(w+V) : ê(U): if supp(w) ç E0, then 6(u,+v) : 6(v) so that (w+g) : (U). For 
w, v G II', if max supp(w) < lllill supp(v) and supp(w)E0 # 0, then a(tv + t,) = 
d(w); so à(w+ ) = d(w). The equations in (*) then follow bv the continuity of 
Vr .  , . .a." = a(g(.,-}). , < p <  and .r G 2;, let 
because & is at most three-to-one.) Thus fiT0] 
and f[Ts] ç Çs if w 5 P < A. 
({ - 
p < A and let (.e,p)  Çp. Ifp < w, let q = p+ 1; otherwise let q = p. Pick 
g'  c( {e • e  E } such that #(j) = . Pi«k .r  Tv. Then ,' + a" + :  Tq and 
/{' +. + ) = ({({{' +  + )).(«{' +. + )),p) 
= 
= (('), (:).) 
= (, ,). 
Whl-«O-,/[d = C. 
The verification that is a holnomorphisln is routine using the equations (,). 
Çhoose  • Ç  T 811d (Dp)<  gua.ranteed by Theorem .5. Since we 
have alreadv observed tha lAI = 2- . a.ll conclusions follow ilnlnediatelv except 
the assertion tha D_ ç K() when A is a successor. To see this recall that 
K() Ç T_ 1 8o tl18 K) 
Theorem 8.5(iv), [Ç-I] 
thus an idem of D and therefore D_ = K(D) 

Corollary 3.17. Let A be an ordinal for which ]AI = c. There exist a set A with 
lAI -- 2 c and an injection g : C.4.) --, IHI such that if q < p < A, y E C e, ad .r Ç C v, 
th« g(a').g(y) = g(a'y), a,,d if p = q+ n for som« n < , tb«n g(y).g(,) = g(y.r). 
Also,  contains a semigroup D = p< D v of idempotetts where for each p < A. 
Dv is a rectangular «onwonent of D. g[Cp] çDv. a,,d the sequen«e (Dp)v<x is 
decreasing in the ordermg of components. For each p < A. [DpI = 2 « and f A is a 
successor, then D_ ç K() ç 

Pro@ Bv Theorem 2.2, NI and NI are topologically and algebraically isomori)hic. 
Also S(U(w)) = c +. So this is an ilnmediate conseqllence of Theoreln 3.16. [] 

It was shown in [5. Corollary 3.4] that there is a _<L-chain (Uo-)o-(tOl of distinct 
idempotents in ON with the property that for each o- < w, U«+l _< 'u«. \Ve are 
uow able to establish a considerably stronger statement. (The necessity in the 
following corolla.ry was also established in [5], but xve includc the short proof for 
completeness.) 

Corollary 3.18. Let  be an ordial. There is a <i-chain (u«)«< of distinct 
idempotents in fin with the lnvperty that for each a < A. l/a+1 __ tic if attd onl'!l 



2810 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

,if lAI _< «. /f lAI _< « ard  is a ,,.«««so. o,.« ,,, cho.»¢ «h a .,«q««¢ ith 
Proof. Necessitv. For each « < A, N* + .« properly contains the compact set 
N* + «+. So one can choose a clopen subsct U« of fin with N* + U«+l ç U« and 
(N* + u«)U«  . The clope subsets of 3N correspond extly to the subsets of 
N ad so there are exactly « of them. 
Suciency. Choose A and 9 as guaranteed bv Corollary 3.17 for A. For each 
p < A, let u v = 9(0.0.p). If A is a successor, then ua-1  9[Ç-] ç DA-I ç 
ç().  
Question 3.19. Is therc a dccreasing 5-chain of idempotents in N* indexed by 
w+ 17 
X close this section by observing thnt it is consistent with ZFÇ that there are 
kempotents in N that are hot members of any notrivial rectangular subsemigroup 
of N. Iadeed, by [6, Theorems 12.19 and 12.29 and Lemma 12.44], Marth's 
Axiom hnplies that there is nn idempotent p  fin such that. whenever q  N, 
r  ,=l«B(t), and p= q+r, one must havep = q = r. In particular. f 
p = p+ q +p, then p = q. 
I can be shown h ZFC that there are idempotents p in  that are strongly 
right maxhnM; .e., the e(luato q + p = p, vth q  N, imples that q = p 
[6. Theorem 9.10]. If p s an dempotet of this kind. p does hot belong to any 
semigrop h  isomorphic o a semgroup of the form 1/ mdess [AI = 1. 
. CHAINS OF RECTANGULAR SEMIGROUPS AS CO-RETRACTS 
It was shown in [10] that certain infinite chains of finitc rectangular senfigroups 
are absolute co-retracts. X prove in this section a shnflar theorem in which the 
the rectangular semigroups are allowed to be infinite. As a consequence, we obtai 
additional semigroups whk.h can bc algebraically embedded in . 
Definition 4.1. Let A = (A}< and B = (B.}< be sequences of sers. Assume 
that each A, bas a designated elcment a and each B bas a designated element 5. 
Suppose also that, for each, < w, either A, = {a} or B = {5}. For each p <  
we define Dp to be the set of pairs of words of the form (a0al -.- av, bvbv-1 • .. bo), 
where ai  Ai and bi  Bi for each i  {0.1 ..... p}. For 0 < A  w, we let 
D.n. = < Dp. We define a semigroup operation on DA.n. as follows: if 
.r = (aoa...ap, bpbv_...bo)  Dp and g = (coc...c,dd_l...do)  D, where 
q  p, then 
xy = (0al-.-p, bp---b+ldd_l-.-d0) 

and 

g3 -- (CoC1 " • • cqaq+l.., ap, bpbp_l .-. bo). 
We leave the verification that the operation is associative to the reader. Observe 
that each Dp is a rectangular semigroup. 
Notice that if A is a set with designated element 0. A0 = {0}. B0 = A. A1 = .4, 
B1 = {0}, and A, = B, = {0} for n > 1, then D.4.., is isomorphic to CA.,» (Send 
(0. a) to a and for p > 0 send theelement (0a00---0,00-.-0b) ofDp to (a, b. p -1).) 
Thus the structure of D.ts., is, in general, considerably more complicated than 
that of 



RECTANGULAR SEMIGROUPS IN STONE-CECH COMPACTIFICATIONS 2811 

Definition 4.2. Let p < w and let x = (o0o 1 "''(lp, bpbp_l ... bo) E Dp. We define 
elelllellt.s 1() and 2(,T) in Dp by çl(X) = (01 "..p-l(tp,pp-1 ""0) and 
2(X) = (GOal ""ap, bp5p-15p-2"'" 50) and if p > O, we define .re and x in Dp-1 
bv xe = (oOal...Op_l, p-lp-2"" o) and x = (o1 ""p-1, bp-lbp-2"'" bo). 
 put u v = (o"" 6p, 5V5V--l''" 0)  D v. 
 show that D is somet.hing like an absolute cretract.. 
Theorem 4.3. Let A = (A)n< ad B = (B)n< be sequences of sets o.s i 
Defiitio .1, let 0 < A  w, artd let D = DA... Let T be a rigt topological 
semigroup, a«td let f " T  D be a sue«tive owmorphism such thot f-l[Dp] is 
copact for eac p < A. Ten tere is a homomorpism g • D  T for wich f  g 
is te ide, tity. ff T is compact atd A < w, ten g[D-l] Ç K(T). 
Proof.  mav assume that A0 = {a0} so that D0 is a right zero semigroup. 
Exactly as in the first paragraph of the proof of Lemma 3.4 we can define g • D0 + T 
such that g is a homomorphism and f  g is the i<lentity on D0. So we assume that 
p > 0 and g has been defincd on q<p Dq. 
For each «  Dp, note that «Dp is a minima.1 right i<leal of Dp ami Dpxis a 
minimal left idem of Dp. So we mav «hoose a minimal right ideal R(«) of f-l[Dp] 
and a minimal left idem L(x) of f-l[Dp] such that f[R(«)] = XDp and fiL(x)] = 
Dp.r. Givcn x e Dp, we have by Lemma 3.3 tha[ g(xe)R(¢(x)) is a minimal right 
ideal of f-[Dp] and L(¢2(x))g(x) is a minimal left ideal of f-l[Dp]. So we may 
define g(x) to be the identitv of the group g(xe)R(¢(x))L(¢2('))g(xr). Notice 
that if T bas a smallest idcal (in particular if T is compact) and A = p + 1, then 
h-(W) ç f-lirai. S+ ç(f-[V]) ç ç(W) .,,d tm g[V]  ç(W). 
+w f(a(«)) e ,.¢(«); = « a,,d f(a(«)) e ;¢(«)' = «. S+ 
is an idempotent in 'DpX and thus f(g(x)) = x. 
Suppose that x e Dp and y e Dq where q ç p. Then ¢(x) = ¢1('Y) and 
ce = (')«. s+ a(«) e (««)(¢1()) ,d a(«)a() e ('e)(¢l(«))(p) ç 
g(xe)R(¢(x)) by Lemma 3.3. Thereforc, g(xy) and g(x)g(y) are members of the 
saine nfinimal right ideal of f- [Dp]. 
If  < , ¢() = ¢() d () = «. s+ («) e z(¢(«))a((:)) 
a(«)a(u) e L(¢(:))(:)a(u)= (¢:(:))a(:)= (¢(-))a((-)). 
If  = p, ¢() = ¢(y) a,,d («) = y.. S («) e (¢:(y))a() 
me minimal left ideal of f-[Dp]. 
By a left-right switch of the al»ove arguments we have that g(yx) and g(y)g(x) 
are in the saine minimal left idem and the saine minimal right idem of f- [Dp]. 
First assume that q < p. Pick a e R(¢(x))L(¢2(x)) such that g(x) = 
g(xe)ag(xç). Thon 

g(x)g(y)g(x) 

= g(xe)ag(x.r)g(y)g(xe)ag(x,) 
= g(xe)ag(x,y.re)og(x,) 
= («)g(xxe)(xT) 
= g(xe)ag(x)g(xe)ag(x,) = g(x)g(x) = g(x). 

So g(x)g(y)g(x)g(y) = g(x)g(y) and g(y)g(x)g(y)g(x) = g(y)g(x) and thus g(x)g(y) 
= g(xy) and g(y)g(x) = g(yx). 
Now assume that q = p. Assume also that Bp = {5p}. (The case that Ap = 
then proceeds by a left-right switch of the following argument.) Then ¢2(x) = 



'2812 NEIL HINDMAN, DONA STRAUSS, AND YEVHEN ZELENYUK 

_( = . A.o . = .« = ._. Th,,, 9(.)9()  L(:(.r))9(x)g(y«) = 
L(up)g(tp_) and g(y)g(xt)  L(up)g(tp_), which is a lninimal lcft ideal of 
f-l[Dp] by Lelllnla 3.3. ' have alleady verified (ha(g(.r)g(ye) alld g(y)g(xe) 
are idempotents. So, since they are idempoten/s in /he saine minimal lefl ideal 
(.)(y«)()(«e) = ()(e). we bave 
(:.)(.q)g(.) = g(..)(,..)(xt.r) 
= (.)g(«)g()(..)g(.) 
= (.)(.'»'t)(.) 
= g(..,»)g(.«) = g()g(.) = g(). 
(',onsequently, g(.r)g(y) aml g(y}g(x) are idempotonts. 
Corollary 4.4. Suppose lhal nis a infi,ite cardiml and lhat eoch 
«ith«r {0} or 2 2. Thon DA.. tan be embedded in. . 
Proof. N»r each p < , we give Dp t]le topology defined by regarding Dp as a 
subspace of (fin 2p+2. We define the topology of D bv ta.king each Dp to be clopen 
in D. Then D is a topohgical semigroup vith a dense subspace of cardinality 
The conclusion then follows from Theorems d.3 and 2.5. 

I EFERENCES 

1. J. Berglund, H. Junghenn, and P. Milnes, Analysis on semigroups, V'iley, N.Y., 1989. MR 
91b:43001 
2. r%V. Çomfort ai,d S. Negrepoiitis, The theory of ultrafilters, Springer-Verlag, Berlin. 1974. MR 
53:135 
3. L. Gillman and NI. Jerison. Rings of contznuous functions, van Nostrand, Princeton, 1960. 
MR 22:6994 
4. N. Hindman and J. Pym. Free groups and semigroups in/3N, Semigroup Forum 30 (1984), 
177-193. MR 86c:22002 
5. N. Hiidmai and D. Strauss, Chams of idempotents in pN, Proc. Amer. Math. Soc. 123 
(1995), 3881-3888. MR 96b:54037 
6. N. Hindman and D. Strauss, Algebra in the Stone-Cech compactification: Theory and appli- 
cations, de Gruyter, Berlin, 1998. MR 99j:54001 
ï. D. McLean, ldempotent semigroups, Amer. Math. Monthlv 61 (1954), 110-113. MR 15:681a 
8. J. Pym, Semigroup structure zn Stone-Cech compactifications, J. London Math. Soc. 36 
(1987), 421-428. MR 89b:54043 
9. W. Ruppert, Rechstopologische Halbgruppen, J. Reine Angew. Math. 261 (1973), 123-133. 
MR 47:6933 
10. Y. Zelenyuk. On subsemigroups of/31 and absolute coretracts, Semigroup Forum 63 (2001), 
457-465. MR 2002f:22005 

FACULTY OF CYBERNETICS. IYIV TAIAS SHEVCHENKO UNIVERSITY. VOLODYMYRSKA STREET 
64. 01033 KYIV, UKRAINE 
E-mail address: grishko@i, com.ua 



TRANSACTIONS OF THE 
AMER1CAN MATHEMA'FICAL SOCIETY 
Volune 355, Nurnber 7, Pages 2813-2828 
S 0002-9947(03)03282-3 
Article electronicMly published on March 12. 2003 

GALOIS GROUPS OF QUANTUM GROUP ACTIONS 
AND REGULARITY OF FIXED-POINT ALGEBRAS 

TAKEHIKO YAMANOUCHI 
Dedicated to Professor Masamichi Takesaki on the occasion of hzs seventieth birthday 

ABSTRACT. It is shown that, for a rninimal and integrable action of a locally 
compact quantum group on a factor, the group of autornorphisrns of the factor 
leaving the fixed-point algebra pointwise invariant is identified with the intrin- 
sic group of the dual quantum group. It is proven also that. for such an action, 
the regularity of the tixed-point algebra is equivalent to the cocommutativity 
of the cluantun group. 

l. INTRODUCTION 

When given an action o of a locallv COnlpact quantum group G on a von Neu- 
mann algebra A, one may associate to it the subgroup Aut (A/A ) of all autonlor- 
phisms of A leaving the fixed-point algebra 4 ' invariant pointwise. Let us call 
this subgroup "the Galois group of a". As [1, Theorem III.3.3] suggests, it would 
sometimes happen (or be expected) that the Galois group carries an important 
piece of information on the quantmn group G itself. With this philosophy in mind. 
we started in [17] to investigate Galois groups of lninimal actions of coinpact Kac 
algebras on factors by making good use of the Galois correspondence established 
by Iztuni, Longo and Popa [7]. In [20], we succeeded in describing the Galois grottp 
of any minimal action of a compact Kac algebra as the so-called intrinsic group 
of the dual discrete Kac algebra. This extended the result of [1] cited above. As 
an application, we were able to show that. if the quantum group in question is 
finite-dimensional, then its cocomlnutativity is equivalent to the regularity of the 
fixed-point algebra. Our main goal of this paper is to extend these results to a 
larger class of locallv compact quantum groups, hOt only compact Kac algebras. If 
we trv to achieve this goal exactlv along the line carried out in [17] and [20], then 
a Galois correspondence for a (lninimal) action of a more general locally coinpact 
quantum group would certainlv be needed. At the moment, it seems that the re- 
sults of Enock in [4] would answer this purpose. Unfortunately, there are however 
a few lnistakes in his proofs, and. to the best of the author's knowledge, they have 
hot been restored yet. So we cannot apply Ellock's Galois correspondence to the 
situation we will consider in this paper. Therefore, we will adopt a new approach 
here that does llot resort to anv Galois correspondence. 

Received by the editors June 24. 2002 and, in revised form. November 6, 2002. 
2000 Mathematics Subject Classification. Primary 46L65; Secondary 22025, 46L10, 81R50. 
hey words and phrases. Locally compact quanturn group, action, factor, regularity. 

@2(103 American Mathematical Society 

213 



2814 TAtxEHIKO YAMANOUCHI 

The outline of this paper is the following. In Section 1, we fix the notation used 
in the whole of our discussion. Basic fa«ts about, locally COlnpaet qllalltUlU groups 
(iii the sense of Kusterlnans and Va.es) aud their actions on von Neumann algebras 
are collected. In Sect.ion 2, we will prove that the Galois group of a milfimal, 
int.egrable action of a locallv compact quantum group G is topologically isomorphic 
to the intrinsic groul) of the dual G. Section 3 is concerned with regularity of the 
fixed-point algebl'a of a lninimal, integrable action. We prove that the regularity 
considerably deterlnines the structure of the qUalltUlll grollp. Naluely it is shown, 
with some exception, that the fixed-point algebra is regular if and only if the locally 
colnpact quantum group under «onsideration is cocomnmtative. In Section 4. we 
lnake a few renmrks on the Izmni-Longo-Popa Galois correspondence. Olm of them 
concerns an explicit formula fol" the inverse lnap of their Galois correspondence. 
Finally, we includc an Appeudix fol" SOlUe auxiliary results which are applied to the 
argument lnade iii Section 3. 
The author is grateful to Professors Michel Enock and Stefaan Vaes for having 
ilfformed hiln that there are mistakes in SOllle proofs iii [4]. He is also indebt.ed to 
Professor Masaki lZUlni for indicating an crror in the earlier draft, of the manuscript. 

2. TERMINOLOGY AND NOTATION 

(àiven a VOll NetlUlallll algebra A and a faithfld llOrlllal semifinite weight ¢ Oll 
A, we introduce the subsets n, rn and rn of A by 

* + = me N A+. 
n = {x  A" ¢(x*x) < oe }, me = rionS, rn, 

The standard (GNS) Hilbert space obt.ailmd frOlll ¢ iS denoted by H. We use 
the sylnbol AO for the canolliCal embedding of ne# into H 0. The modular objects 
such as the lnodular operator, the lnodular conjugation, the S-operator, the F- 
operator, t.he modular automorphism group, etc. associated t.o çb are denoted bv 
Ve, 34,, S¢, F¢, a ¢, respectively. (Since we follow the notation enlployed in [10], 
the sylnbol Vwill be used t.o deuote the lnodular operator of a weight.) The set of 
unitaries iii A is denoted by H(A). For a von Neulnallll subalgebra B of A. define 
JV'(B) := {u  H(A) : uBu* = B} and call it. the normalizer (group) of B in A. 
V:e let B(H) stand for the algebra of ail bounded operators on a Hilbert space 
H. 

2.1. Locally compact quantuln groups. 
lï)efinition 2.1. Following [10] (see [9] also), we sa,v t.hat, a quadruple G = (]il, 
A, 0, b) is a locally compact quantum group (in the von Neulnann algebra 
sett.ing) or a von Neumann algebraic quantum group if 
(1) /11 is a von Neumann algebra; 
(2) A is a unital normal injective .-homomorphism from M into M ® M sat- 
isfying (A ® id) o A = (id ® A) o A: 
(3) o is a faithful nornml selnifildte weight on kl such that 
ri1+ - 
o((w®zd)(A(:r)))=o(z)w(1) (VwM. +, V:e ), 
(4) /, is a faithful normal semifinite weight Oll Al such that. 
/,((id ' w)(A(:r))) = #,(a:)(1) (Vw • M. +, Vin • më). 



GALOIS GIqOUPS 2815 

Let 118 fixa localh, compact qtlalltllln grotlp Ça; : (]ll, A. ç, b) throughout the 
rest of this section. We will always think of M as represented on the GNS-Hilbert 
space H obtailied ffoto . By the left invariance of 
on H  H characterized by 
II'(G)*(A(x) @ A(y)) = A$(A(g)(,r @ 1)) (x,g  n). 
This unitary is called the Kac-Take.soki operator of G, and is denoted silnply by 
Il if there is no danger of confllsion. The modular operator and the lnodular 
conjugation of  will be denoted simply by Ç a.nd J. The unitarg atipode, the 
scalig gronp and the .scalig co.tmd of  are respectively denoted bv R, {7t}tER, 
p() 0). As iii [10], wç aS81llll that @=ço R. 
According to [10]. there canolficallv exists another locally compact quantmn 
group â = (kî, , , ), called the locally compact quaztum group dual to a such 
that {M. H} is a standard lepl'esentation. So we always regard 
In fact. I is by definition the von Nenmanl algebra gcncrated by {(w @ id)(ll ) • 
w  M,}. The lnapping : w  M,  (w @ id)(lI')  I is called the Icft regular 
repre.settation of G. There is a canolfical identification (= the Fourier trllsforlll) of 
Hç with H. So we conskler thc lnodula.r operator and the lllOdtllar conjugation of 
, denoted by Ç and ,Ï, as acting on H. The mfita.ry a.ltipode, the scaling group 
and the lnodular elelnelg of  are denoted respectively by , , .  say that 
G is COlnpact if (1) < . In this case, we agree to take  to be a state. X say 
that G is discrete if G is conlpact. For the defillitions of locally compact qllalltUln 
groups such as the COllllllllta.llt t, the opposite G °p etc., we refer the readers to 
[10, SectiOll 4]. 
It is known that every locally colnpact group F canollically gives rise fo a COlll- 
nmtative locally conlpact qtlalltlllll group whose tlnderlyillg VOll Netunalln algebra 
is L(F). XVe denote it by G(F). The underlying von Neulnalm algebra of the 
dual G(F) is the group von Neunmnn algebra of F gelmrat.ed bv the lefl regular 
repl'esentation of F. 
 denote by /G(G) the set of all unitaries u  I satisfying &(u) = u @ u. 
The group IG(G) is called the intrinsic group of G. Next defille Ç(G) to be the 
group ofall automorphisms fl of I such that (fl@id) o& = off. By [2] (see 
[19] also), /G(G) is topologically isolnorphi« to Ç(G) through the mapping: v 
IG()  fl := Ad v[  Ç(G). Here IG() is endowed with the strong-operator 
topology, and, for a general (separable) von Neumalm algebra P, we consider on 
the automorphisln group Aut (P) of P the topology of silnple COllVergellce Oll the 
predual. It is kllOWll that v is the canonical implelnentation of fit,. 
ç soEv that N Ç I is a right co-ideal (von Nemnalm subalgebra) of G if 
is a von Neumann subalgebra of I satisfying &(N) ç N @ ,I. Thanks to [4, 
Théorèlne 3.3], we know that N Ç I is a right co-ideal of G if and only if Olle has 
(2.1) N = Ma(MaN')'. 
A (le) action of G on a von Neunmlm algebra A is a normal injective unital 
• -homolnorphism a from A into kI @ A satis'ing (id 
([11). 
Fix an action a of G on a von Netllllllll algebra A. By [16. Proposition 1.3], the 
equation 
%(.):=($ia)((a)) (.+) 



2816 TAKEHIKO YAMANOUCHI 

defines a faithflfi normal operator-vahmd weight T, fl'OIll 4 onto .4  := {a E 4 - 
c(a) = 1 ® a}, the fixed-point algebra A  of c. \Ve call T the operator-valued 
weight associated to the action o. 
The ClOssed produ«t of A bv the action  is by definition the von Neulnann 
algebra generated bv c@4) and ]Î  c. We denote it. bv G « < A. By [16. Propo- 
sitioli 2.2]. there exists a unique action a of op on G a < A. called the dual action 
of a, such that 
(G«< .4) a = n(A), &(z ( 1) = A°V(z) ® 1 (z e M). 

For everv faithful normal semifinite weight b on A. by using the operator-valued 
weight T associated to the duai action d, lhe equation 

By [16, Theoreln 2.6], therc is a unital ,-isomorphisln O from the double crossed 
l)roduct (°Pa < (Ge < A) onto B(H)® A. and an action d of G on B(H)® 4 
such that 

g := Ad (El'*2 ® 1) o (er  id4) o (id(B,,) ® a). 
(Ad (Jj) ® 6) ) o  = d o 6). 
where 1" := (J ® J)EII'*z(.Ï  .Ï) and : H, ® H --, H v 0¢ H is the flip. \' call 
à the stabilizatio of a. 
We say that the a.çtion a is integrable if T is semifinite. The action a, is said to 
be minimal if AV/(A) ' = C and the linear span of {(id"3ca)(a(a)) : o  A. ,,_, G A,} 
is o--weakly dense iii M. 
Finally, G is called a Kac algebra if ri = id a.nd er4' = aV. For the general theory 
of Içac algel)ras, refer to [6]. 

3. REALIZATION OF INTRINSIC GROUPS IN AUT (A/A ) 
Given a von Nelllllallll algebra P alld a von 1Neulnallll subalgebra Q of P, we 
define Aut (P/Q) t.o be the group of ail automorphisms of P leaving Q invariant 
pointwise. 
Let  = (M, A, ç, ) be a locally compact quantuln group. Suppose now that 
we have an action a of G on a von Neulnanl algebra ,4. Throughout this paper. A 
is alwavs assumed t.o be a non-type In factor (n  N). 
The lnapping considered in the following proposition is essentially observed bv 
Enock and Schwarlz in [5] as a special case of their constructions of certain lnor- 
phislns associated to an action of a Kac algebra. The lnapping is still defined even 
for a general locallv compact quantum group. 



GALOIS GROUPS 2817 

Proposition ;3.1. There e.rists a "uniqtte co,ttinuous hontomorphism front Ç(G) i'rtto 
Aut (A/A ) su«h that, with O the image of fl  ç(G) under this homomorphism, 
(J  d) o., =  o 
If the action o enoys the propert that the linear span of {(idM  w)((a)) : a G 
A, w G .4,} is ¢-'u,eaklp dense i .I. the the above homomorphism is ijective. 
Poof. net fl ç(G) a,,d ,, e .-. Se X := (d  i.«)  () e t  4. Then 
(idM  )(X) = (idM  ) o (fl  idA) o (o) 

= (d @ id^t 
= (13 @ id^t 
= (A®ida 

® id4) o (id^l ® ) o o(a) 
® idA) o (A ® idA)  c(a) 
) o (fl ® ia «) o (,,) = (A  id 

From [16. Theorem 2.7], there exists a ,inique element 03(o ) 
(3.1) (fie id4) o (a) = X = (0fl(o)). 
It is easv to (']w«k bv using (3.1) Hmt 03 is a hOmOlnorphism rioto A into itself. 
Moreover, one tan easilv verify that 0d,& = 0d, o 0& and Oid = id. Hen«e 0 is 
ail automol'phism of .4. That the restriction of 0 d to .4" is the identitv fo]]ows 
also from (3.1). Thus the lnapping fl G Ç(G)  00 G Aut (A/A") is indeed a 
homomorphism. Be«ause of (3.1), we find that (fl id4)[(a) is an autolnOllflfiSln. 
With this in lnind. 0, bas the form 
0. = - o (fl 
Henee fl  0d is continuous. 
Now suppose that the linea.r si)an of {(idt OE )(o(a)) :  G A.  G A.} is 
«-weakly dense in M. If 0 = id. then (3.1) implies that fl is the identitv on 
{(idt  )((a)) :a G A.  G A.}. So  = id. Consequently. the homomorphism 
in question is injeetive. 
Lenmm 3.2. Let fl be an automorphism of 3I such that there is a 0 G Aut (A) 
such that (fl @id) =  0. Suppose that the lmear san of {(idat  ¢)(((a)) : 
a G A.  G A.} is -weakly dense in M. The fl belongs to Ç(G). ad one bas 
0 = 
Proof. Note first that, if 0 is an automorphisln s al)ove, then it automatically 
belongs to Aut (A/A'). Therefore, it suffices l)v Proposition 3.1 to show tlmt fl 
belongs to Ç(G). 
Let a G A and w E A.. Then we have 

03 0 id^ ) o /-X( ( id^l 

/3 satisfies (/3 ® id) o A = A o 3 



2818 TAKEHIKO YAMANOUCHI 

Let  be the trivial action of G on C. Namely, t is the mapping frOlll Cinto 
bI®C defined by,(c) := l@c (c ff C). Then the crossed product G, C is 
(canonically isomorphic to) Î, and the duM action i is the coproduct o. It is 
also clem- that the dual weight of tc corresponds to . In this ce. the stabilization 
ï of t h the form ï(x) = EV*E(1 @ x)EVE for 811y X  B(Hw). Therefore, we 
obt, ain 
(3.2) 6(x  1) =/(x) , 1 
for any x  B( H¢). 
Lemlna3.3. Let z be in B(H¢) such that z@ l G« A. Then z belon9s to M. 
Pro@ Let z be an element as above. Since (B(H¢) @ A) 6 = G«  A. we have 
6(z@l) = l@z@l. Onl,eotherhal,d. by (3.2), wehave6(z@l) =[(z)@l. 
Hence we obtain [(z) = 1 @ z. Since B(H)  = M, z must belong to .I.  
Lemlna 3.4. Suppose that  is minimal. Then we bave (A)'  B(H) @ A = 
M' @ C. 
Pvqf. It is clcm" tha M' @ C is contained in a(A)'  B(Hç) @ A. Take anv 
T  a(A)' B(H)@ A. Since T particularly commutes with any element of 
a,(A ) = C@ A a, it follows from the minimality of a that T belongs to B(Hç) C. 
So it has the form T = y @ 1 for some y  B(Hç). If o ff A and w ff A., then we 
bave 
y(id@w)(a(a)) = (idvw)(Ta(a))= (id@w)(a(a)T)= (id@w)(a(a))y. 
By minimality of a, y nmst be in M'.  
Since (Ga  A)  = a(A), it follows from Proposition 3.1 that there exists a 
contimous homomorphism fl e Ç(o)  O  Aut (Ge  A/a(A)) satisb-ing 
( @ id) o  = & o 0. Since & enjoys the property mentioned in Proposition 3.1. 
the homomorphism fl  0d is injective in this case. 

Lemma 3.5. If the action 
O • Aut (G«< A/a(A)) is 

Pro@ Let 0 be in Aut (G « cE 
O(z® 1)(1®b)= 

is minimal, then the homomorphism fl ff Ç(G °p)  
topological isomorphism. 
A/c(A)). If z E M and b  A . then 
O(z® 1)a(b)=O((z® 1)a(b))=O(z®b) 
O(c(b)(z® 1)) = (1 ®b)O(z® 1). 

This shows that O(M ® C) __Ç_ (C ® A«) ' = B(H¢) ® (A«) '. From this, we obtain 
O(M 6; C) ç B(Hc) ® {A C (A«) '} = B(Hc) ® C. 
Hence, for any z  M, there is a torique flo(z)  B(H) such that 
(3.3) O( z ® 1) = flo( z) ® 1. 
Thanks to Lenlma 3.3, flo(z) belongs to BI. Due to (3.3), it is easy to see tllat ff0 
is ail automorphism of M, and that flolo2 = flOl o flo2, flia = id. 
If a ff A, then 
(3.4) (flo®id)(&(a(a)))=(Bo®id)(l®a(a))= 1 ®a(o)=d(O(a(a))). 



GALOIS GROUPS 2819 

Fix a faithful nornm.1 semifinite weight w on A, and regard A as represented on 
H. Let .] be the modulm conjugation of the dual weight & and U the canonical 
inlplemeltatiol of o on H @ Hw. So U = ,](,1 @ ,Jw). ('hoose the canonical 
inlplenmntatiol unitary I 
COlmnUtes with ,L, we also have 
It follows front Lemma 3.4 that there exists a unitary v  I' such that I = v @ 1. 
Therefore we have flo(Z) = vzv* for any z Ç ;[. By [12, Proposition 1.9], which is 
valid also for any locally COlnpact quantum gl'OUp, we sec that rt' belollgS to IG(') 
for some r  C with ]r = 1. Since Ioth I" and .ri" are the canonical ilnplementation 
of 0, we lnust have r= 1. So , is in IG(g'). It follows tiret 0 belongs to çP). 
Let z  M. Then we have 
a(0(z @ 1)) = (l)o(z) @ 1) = °P(o(z)) @ 1 
= (/o  id) ()  l = (o .id) d(» ). 
From this, together with (3.4), we get (ff0 @ id) o d = d o 0. By Lennna 3.2. we 
find that 0 = 0 0. Thns we have showll the SUljeclivily. The inverse lnap is also 
«ontinuous due to (3.3). 
Theoreln 3.6. If  is a minimal, integrable action of a locally compact quantum 
group  = (BI. &. ç, ¢) on a factor A. then there exists a topologicai isomorphsm 
Ç e Ç(g)  0 e Aut (A/A ) with the property ( @ id) o  = a o 0. 
Pro@ If .4  is infinite, then, by [16, Proposition 6.4], a is a dual action. Hence 
the assertion follows frOlll Lelnlna 3.5. 
To deal with a general case, take a (separable) infinite factor L, and put , := 
L @ A. Also definc d := (« @ id) o (idL @ a), which is an action of  on ,4 with 
A a = L@A  infinite. Remark that aut (/2 a) = {id eO. 0 e aut (A/A")}. Let 
0  Aut (A/A). By the previous paragraph and the above renmrk, there exists 
13 e çg) such that (floEid)o& = 6o(idLO). But this yields 

4. I={.EGULARITY OF A  IN .4 
AS ill the previous section, let cî be a minimal action of a locally colnpact quan- 
tum group Q = (AI. . ç, ) on a factor A. 
 represent A on a (separable) Hilbert spe K so that {A. K. OE4 } is a standard 
representation. Let u G (A). Then the restl'iction of Ad u to (A)  clearly defines 
an autolnorphism 0u in Aut ((A)'/A'). The holllOlllorphislll u Ç (A )  Ou Ç 
Aut ((A)'/A ') obviouslv has H(A ) as its kernel. The bic extension for 4  ç A 
is denoted by Al. So we lmve A = OE4(A)J4. If A  is infinite, then so is A. 
Thanks to [3, Corollaire 1], we may then choose the above Hilbert space K in such 
a way that there is a unit vector 0 G K that is cyclic and separating for both A 
and A . Let OE4 then denote the lnodular conjugation of A associated to 0- 
Lemma 4.1. Suppose as above that A  is infinite. Then the homomorphism 
u e (A )  0  Aut ((A)'/A ') defined above is surjective, that is. it bas 
Aut((A)'/A ') as its image. 
Pro@ Let 0 be in Aut ((A)'/A'). Since {(A) ', K} is a standard representation, 
there exists a unitary v on K such that 0 = Adv(A),. Since OA, = id, v belongs 
to (A')' = A. It is eass" to see that c is in (A).  clearly have 0 = 0.  



2820 TAKEHIKO YAMANOUCH[ 

Remark. ff A is finite and G is finite-dilnensional (so that the .bines index fA " A 
is filfite), then it follows ff'oto [11, Proposition 1.7] that the lllap   (A 
0 e Aut ((A)'/A ') is smjective. 
From now on, we asstnne that a is miuimal and itttegrable (A is hot necessarily 
infinite). Fix a faithful normal semifinite weight « on A. and represent A on 
H now. Let U be the canonical ilntflementation of a associated to «. Due to 
[16, Proposition 6.2], this assmnl»tion is equivalent to the one that ( is outer and 
imegrable. From [16, Corollaly 5.6] and [16, Ploposition 6.2], it follows that the 
inchtsions 
C @ A  ç a(A) ç G.4 and ,4  ç A ç A 
are isomorphic. According to [16. Corollm'y 5.6], the isomorphism p" G a A 
is characterized by 
(4.1) p(a(o)) = o (a e A), 
(4.2) p((ç@id)(ll)@l)=(ç@id)(U*) (ç3I.). 
lncideltally. Equatiou (4.2) can be rewritten as 
(4.3 p('(ç)  1) = (3Jd3  id)(U) (ç  (BI').), 
where A' stands for the lefl regular representation of G  that is given by A() 
A(3.]ç* J,Ï)*.  will make use of this isomorphism in the discussion that follows. 
' = 3.4 3, Aut ((W*)'/A ') is isolnorphic to Aut (G 
Since A = J(A ) J and A ' 
A/(A)). Thus we obtain a hon,on,orl,hism ff'on, (A ) into Aut (G  
Bv using the isomorphism p introduced above, it is explicitly given as follows: 
u  (A (*)  0 := p- o Ad (JuJ.)]A o p. As we saw in Lelnlna 4.1 and the 
remark afler it, this holnomorphism is smjective if A a is infinite or if A is finite 
and G is finite-dimensional. But it may hot be in general. So our next goal is to 
identifv its image in detail. For this, first note that. thanks to Proposition 3.5, it is 
enough to identifv automorphisms fl  Ç(G °p such that 03 = 0 for , 
Moreover. since each 0 has the form 0 = Ad (v @ 1) for a torique v  IG(G ) with 
d = Ad v, it suces to identify unitaries v  IG(G') such that Ad (v @ 1) = 0 for 
some u e W(A ). 
Proposition 4.2. Let o be a miaimal aud int«grable action of a locall 9 compact 
quantum gro'up G = (I. A. ç, ,) ou a .factor A. 
(1) For a, 9 u  .ç(A). there ezists a uaique uaitary w(u)  IG(G) such that 
() F    (A), ,it w() i t (), , 
O, = Ad(Jw(u)J  1) = Ad(J N J)a(u)(1N u*)(J N J). 
We deaote the uitary Jw()J la IG(G') b9 v(u). 
Pro@ Let u  (W). 
(1) It is straightforward to check that a(u)(1  u*) comlnutes with any elelnent of 
the forln lça, where a  A . So it is contained in BI'Ao(COEAa)  = IC. Hence 
there exists aunitarv w  BI such that n(u)(1 )u*) = w@l. Thus a(u) = w@u. By 
applying A @ id to both sides of this identity, we obtain A(w) = w  w. Therefore 
w belongs to IG(G). So put w() := w. 



GALOIS GROUPS 2821 

(2) Choose/3 E 9(G °v) a.nd t, E IG(G') such that 0u = 03 and  = Ad v. Since 
G' = G °v, we have (A'(0)) = A'(v0). Therefore, by (4.3), the identitv 0 = 0O is 
equivalent to the lleXt: 
Ad ( 3& ) ( ,lO3 J  id)(U) = (.3vO&ï  id)(U) 
This is flrther equivalent to 
(,Li  &,,L)U(.L)  ,LP,L) = (3J  )U(,Li«  ). 
By using ff* = (j e &)U(J e ,L), we can rednçe the al,ove identitv to 
«(u)(1 e u*) = ,Je.] e 1. 
Consequently, we obtain «(u)= ,lv,l. 
Definition 4.3. Let /3 be an action of G on a von Nemnalm algebra P. For a. 
right «o-ideal N of G, the intermediato von Neulnam snbalgebra P(N) of P ç P 
associated to N (sec [7]) is defined by 
The fi)llowing lenllna is In'OVen in [4] for the case wh«re G is a. Woronowicz 
algebra. The clailn and its pl'oof are still valid even for a general locallv compact 
qua.nt mn group. 
Lemlna 4.4 (Proposition 3.5, [4]). Let N be a right co-ideal of G. Tken we kove 
Pro@ Denote by Nt the right-lmnd side of the al)ove clailn. Clearl) we have 
NI ç N. Let z  N. For any 9  N, p  B(H). and w  31., we have 
(p @ w)((  1)(z) = p(9(id  w)((.r))) 
= p((id @ w)((«))9)= (P @ w')((z(9  1)). 
This shows that (z) is in N1 N M. Hence (N) ç N N 31. In particular. Ar is 
also a right co-ideal of G. Therefi)re,  := [ defines a.n action  of G  on 
If z  N. then, by the above result, op(.r)  [ @ Nt. Moreover, we have 
(i«  )(;(«))= (i«  Z;)(5;('))= (Z ;  
From [16, Theorem 2.7], it follows that (z) belongs to 7(N) = (A). Thus 
.r Ç N 1 . 
Lemma 4.5. Let  be an action of G on a von Nemann algebra P. For 
intenediate von Nemnann subalgebra Q 4 pa ç p. {(id @ ')((z)) • " Ç Q.  
P, }" is a right co-ideal 4G. We denote this rigkt co-ideal bg NA(Q). md call 
the right co-ideal associated to Q. 
Pro4 Let 9 e &.(Q)'. For any p,  e B(H), w e P. and ze Q. we have 



2822 TAKEHIKO YAMANOUCHI 

From this, we sec that A((zd®a)(/3(x))) is inch,ded in N(Q)® M. Hence N3(Q) 
is a right co-ideal of G. [] 

Lemma 4.6. Let 13 be an action of G on, a von Neumann algebra P. |Vith the 
notation introduc«d above, we bave N = N((GOE P)(N)) for any rigbt co-ideal 
N of P. If L is the von Neumann subalgebra of   OE P generated by (P) and 
N @ C for some rigbt co-ideal N ofG p, then N(L) = N. 
Proof. Let N be a right co-ideal of p. It is plain to see that ((G P)(N)) is 
contailmd in N. Let x  N alM p  B(Hç),. Since À°P(x)  N @ M. we have 
(.,.  )= A»(.)  1 e  (g  P). 
Tlms x@ 1 belongs to (GOE P)(N). lfw  P, is a state, then 
(id @ p)(P(x)) = (id @ p@)(P(x) @ 1) 
= (idpw)(Ç(« 1))  N(( 
It follows ff'oto Lemma 4.4 that N is «ontained in OE)((G P)(A')). Therefore, we 
bave prove that N= (( OE P)(N)). 
Let L be as above. Take any st.are   A.. Then, by Lemma 4.4, we hare 
,ç = {(id e )(%)) " « e , e ,}" 
= {(id¢)(h(.) 1)« e »'. e }" 
= {(d  ¢)((,  1))' e X,e a1%}" ç (). 
In the meantime, L is clearly included in (G  OE P)(N). Hence, bv t he result of the 
previous paragraph, we get N(L) Ç N((OE P)(')) = N. 

Lelnma 4.7. Let G = (M. A. ç, ¢) be a locallg compact quantum group. Then 
1"I 9(G) = C if and only if G is coromtalive. 

Proof. If G is commutative, then we clearlv bave ]I ç(G) = C. 
Put N := IG()'. It is easy to see tha.t N is a two-sided co-ideal (von Neumann 
subalgebra) of .Î (more precisely, of ). Moreover, we have M ç(G) = /I V/N'. 
From [4, Théorèlne 3.3] (which is still valid for a locally compact quantum group), 
it follows that N =/Î if/lç(G) = c. Then Ç is cocommutative. [] 

As explained in Section 1, the mapping v  IG(G')  Ad vl ï  ç,op) is 
a topological isomorphism. Let /3,.  ç(G °p) stand for the inmge of v  IG(G') 
under this isomorphism. 

Theorem 4.8. Let c be a minimal and integrable action, of a locally compact quan- 
tum group G = (M,A,ç,,) on, a factor A. ||e set P := .Af(A) '' and define 
p 
P to be the basic extension of P C_ A, i.e, P := J, J. Itïth the isomor- 
phism p" GOEA  A, put Q := p-(P). Let w(-)'N'(A )  IG(G) and 
v(-): .N'(A )  IG(G') be the maps obtained in, Proposition .2. 
(1) The maps w(.), v(.) are continuous homomorphisros with H(A ) their ker- 



GALOIS GROUPS 2823 

(2) 

(3) 
Proof. 
() 

We bave 
Q! = (G OEA) {0:ue'h/(A«)} = (G OEA) {0t-(}: 
= (G  A)(ï{z( :ev()} ). 

(1) This is straightforward. 
With the original dcfinition o[ 0, ( 

E JV'(A")), we clearlv have that 

(A) {°:ueN(A)} = Pl. So we get the first equality of our assertion. Bv Propo- 
sition 4.2 (2), we llave 0, = t3,(u), which yields the second equality. The la.st 
equality follows from the fact that 0o alwavs satisfies (/3 OE id) o & = & o 0 due to 
Proposition 3.1. 
(3) This follows fronl Part (2) and Lemma 4.6. 
Let P Ç Q be an inclusion of von Nemnmm algebras. If the normalizer group 
(P) of P in Q generates Q, we sv that P is rc9ular iii 
The next theoronl is a direct generalization of [20, Theorenl 3.6], where we treated 
only the case where  is finite-dimensional. 
Theorem 4.9. Suppose that a is a minimal and inte9rable action of a locally 
compact quantum group G = ( M. , ç,  ) on a factor A. 
(1) If A' is regular in A, rb en G is cocom.mutative. 
(2) If .4  is inflnite, or if A is finite ad G is finite-dimen»ional, then the 
cocommutativity of G implies the regvlarity of A' in A. 
Pw@ Retain the notation elnployed iii Theorem 4.8. 
(1) Suppose that A  is regular in A. Then Q = (A). By Theorem 4.8 (3), we 
get 
ï {,,() •. e (A)}' = {.,:(4} = C. 
In particular. M  {w(u) "   (A)} ' = C, because wc bave J3lJ = M in 
general (sec [101). Sin«e {«(u) • u e (A)} '' is a twsided co-ideal of ç, it 
follows from (2.1) that 
Hence  is cocommutative. 
(2) Assume that A  is infinite, or that A is finite and  is finite-dilnensional. 
Let v  IG('). It follows ffoln the proof of Lemma 3.5 that Ad (v OE 1) is in 
Aut ( oE A/a(A)). Bv assullption, it also follows ri'oto Proposition 4.2 and the dis- 
cussion preceding it that u e (A )  0 = Ad (v(u)OE 1) e Aut (ç A/a(A)) 
is an isomorphism. Hence there is a unique u  (A a) such that 
Ad (v OE 1) = 0 = Ad (v(u) OE 1). 
By t.he proof of Lemlna 3.5 again, we nmst have v = v(u). Therefore. the map v(-) is 
surjective, i.e., v((A)) = IG(ç'). So, if  is coconlmutative, then v((A)) ' = 
BI. Fron this, it follows that 
 {z():e(A')} = ;ï M = C. 
By Theorem 4.8 (3), Na(Q) = C. This means that Q1 = (OEA) a = a(A). 
Therefore A = P. 



2824 TAKEHIKO YAMANOUCHI 

Next we would like to discuss Theoreln 4.9 (2) iii the case where A is finite and 
G is infilfite-dilnensional. 
Let F be a (countable) discrete group and  be a minimal co-action of F (i.e.. a 
niinimal action of G(F)) Oll a type II factor A. For any 7  F, define 
and call it the eigensubspace of 7- The subspaces {A(7)}er played a vital part 
in defining the Connes spectrmn F(a) of a in [18]. 
Proposition 4.10. Let F, ( and A be as above. For any 7  F, tbe eigensubspace 
A"(7 ) contains a ¢nitary. 
Pro@ By [18, Theorem 3.17], A(7)*A'(7 ) is a-weakly dense in A «, so that A(?) 
contains t)lenty of nonzer() elements for any   F. Fix an arbitrary 7  F  {e}. 
Put 
B:= ).. ",bA «, X.]'.4(7 . 
Bv using *he minimality of a and *he fac* tlmt A'  A(?) = {0} (V7 # e), one 
can easily w'riD tlmt B is a sui)factor of k&(A) = A  a&(C). Accordingly. the 
torique tracial state on A @ a&(C) restricts to that of B. So the projections 
[(1) a]' [Ô ï] 
m'e equivalent in B, since they have equal traces there. Hence there exists an 
isometry }"  A(?). Since A is finite. Y is a mfitary. 

Theorem 4.11. Suppose that c is a minimal, integrable action of a cocommutative 
locallg compact quantum 9roup G = (5I. X,,) on a II factor ,4. Then .4  is 
're9ular in A. 

Proof. Since A is finite, it follows that G nmst be of COlnpact type. Hence G has the 
form G : ((F) for a unique (countable) discrete group F. Bv Proposition 4.10. ev- 
er)" eigensubspace A'(7) contains a unitary I'(7). Cearly V(7) belongs to 
So it relnains to show that {l'(7)}er and A' together generate A. But. this follows 
flore the next two facts: (i) A is generated by {A*(7)}er: (ii) A'(-),) has the form 
A'(7) = A«I;(7) for any 7  F. [] 

Remark. We remark that, fol" a minimal, integrable action c on an infinite factor A 
with A ¢* finite, the cocommutativity of G does NOT in general imply the regularity 
of A . In fact, suppose that A is a factor of type IIIa (0 < ,k < 1). Take a faithful 
normal state w on A with  = idA, where T := -2rr/log A. We regard this inodular 
action as ail action ct of the one-dimensional torus T on A. It is well known that ct is 
a minimal (integrable) action. Note that A , the centralizer of w. is a factor of type 
Iii. Let u be in A/'(A). It is easy to see that u*cz (u) lies in (A")'çA = C. It follows 
that u belongs to some spectral subspace A(n) := {a • A : c,(a) = za (Vz • T)} 
(n • Z) of the action c. But, according to [14, Lemma 1.6], every spectral subspace 
A"(n) except A"(0) = A  contains no unit ary. This shows that A/'(A ) is contained 
in A ". Therefore A  is hot regular in A. 



GALOIS GIROUPS 2825 

5. IEMAFIIxS ON ILP's GALOIS COFIFIESPONDENCE 

Let et be an action of a conlpact Kac algebra G = (11I, A, ç, g) on a factor ,4. 
In what follows, the action c is aSSUlned t.o ho lninilnal and imegrable, but we de 
net necessarilv assmne that ,4" is infinite. 
In [7], a complete Galois correspondelce fol a minimal action of a compact Kac 
algebra is obt.ained. According te [7, Theoreln 4.4], the map N  A(N) gives a 
one-to-one correspondelwe between the lattice of right co-ideals of G and that of 
interlnediate snbfac'tors of A  C A. 

Proposition 5.1. The inverse map of ILP's Galois correspotdetce cited above is 
given by B  N,(B). 

Pro@ Let L, .51 and 6 be as in the proof of Theol-eln 3.6. Then it is easy te check 
that. for any iltellnediate subfactor B of 4  C_ ,4 and any right co-ideal N of , 
one has 

(5.1) 

.(N) = L ® 4(N), ](L ® B) = 

(Note that ally intermediate slll)factor C of A' _C  has the foliii Ç = L ® B fol a 
Ulfique/3 as above, thanks te [7, Theorem 3.91 and (the proof of) [15. Lelllllla 2.1].) 
Therefore, by considering .21 instead of A itself if necessary, we lnay assulnc ff'oin 
the outset that A  is infinite. Then o is dominant by [17, Theoleln 2.'2] or [16. 
Proposition 6.4]. Se there exists an outer action  of Ç' on .4 ¢ such that {.4, o} 
is conjugate te {Q't< .4 `,/}. Hence we assulne that A = Q't < .4 « g:/ll(| Or = [?. 
First, by Lelnma 4.6, N = N(A(N)) for any right co-ideal N of . 
Let B be an intermediate subfactor of A' C A. Choose the unique right co-idcal 
N such that B = A(Nt) by using [7]. By the result of the preceding paragraph, we 
obtain A(N(B)) = 4(_N,(A(_N))) = A(]V) = B. This completes the proof. [] 

The following proposition is regarded as an extension of [211, Theoren 3.5], where 
we discussed the case of G being finite-dilnelsional. 

Proposition 5.2. Suppose that a is a minimal action of a compact Kac algebra 
G = (/11, A,ç,¢) on a factor A with .4 ' inflnite. Then/NÇ(JV'(A) '') = IG(G)". 
i.e., the right co-ideal erg correspondin9 te the intermediate subfactor JV'(A')" is 
Ia(C)". 
Pro@ Let P be the basic ext.elsion ofthe inclusion P := ,V'(A')" C_ 4. With p the 
isomorphisln introduced just after Lemnm 4.1, set Q := p-1 (p). Bv Lemnm 4.1 
and Theorein 4.8, we have Ï ç(ê°) = Na(Q). Frolll this and Corollary 6.3. we 
obtain Ï ç(-°) =/)(ÏN(P)' ). In other vords, ÏIG(G')' =/)(ÏN(P)' ). 
Hence, by (2.1), we have 
N.(P) = AI ç (_î f N.(P)' )' = ]Il  {¢(-Ï ç IG(G')' )' 
= M ç (M ç J IG(G')' J )'= M  (M ç IG(G)')' = IG(G)". 

Thus we are done. 

Rernark. The above proposition may be used in order te prove Theoreln 4.9 in the 
case of colnpact Kac algebra actions. 



2826 TAIEH IKO YAMANOUCHI 

6. APPENDIX 

Let. a, be a minimal and integrable action of a locally compact quantmn group 
G = (M, A, , ,) on a factor A. We fix a faithflfl normal semifinite weight w on A 
and regard A as acting on H. Denote b.v 3 the modular conjugation of the dual 
weight &. The canonical implementation U of ca, associated to w is then given by 
U = 3,,(.] ® J,,). The basic extension of A 
Let. p be the isomorphism p- G,<A  A that appeared in Section 3. Identitv 
(4.2) can be rewritten in the form: 

(G.1) 

p(A(¢) ® l) : (¢® id)(U*) 

Let. A2 1)e the basic extension of A _C At. Since B(H¢)® ,4 is the basic extension of 
c(A) C G <A. the al»ove isomorphism p can be extended to the isomorphism, still 
denoted by p. from B(H)A onto A.2. Since B(H)A(CA) ' = B(H)C C, 
the equation 
defines a ,-isomorphism il ff'oto B(H«) onto 42  (A') '. Since B(H)  4  
(A)' = M'  C and G  A  (C  A«) ' = M  C, it follows from (6.1) (recall that 
(6.2) A2 n A' = H(3I'). .4, n(A")'=H(î)={(N.id)(U*).ÇM.}". 
Since kA(ç))= .lA(ç)*J = A(o R), it follows from (6.1) that 
(6.3) H((:)) = &H(z)*,L (z e lÎ). 

Lemma 6.1 (Proposition 4.4. [4]). For any interrnediate subfactor B of ,4   A. 
N.()' ®C = B(H) ® A  ()'. 
Equiva le, tlg. H (IV (B)') = A._ V B'. 

Pro@ Since B(H)®Aç(C®.4)'= B(H)®C, wesee that B(H)®Aça(B)'= 
{T ® 1: T Ç B(H), T ® 1 e c(B)' }. For T e B(H), we have 
r®lea(B)' <==> (\®¢)((r®l)a(b))=(\®¢)(a(b)(T®l)) 
( e B(H,)., ¢ e .4.) 
 \(T(id®¢)(a(b)))= \((id®¢)(a(b))T) 
(\" e B(H,)., ¢ • A.). 
The last condition is equivalent to T being in N(B)'. [] 

Lemma 6.2 (Corollaire 4.5, [4]). Let B be an intermediate subfactor of.4  C_ A. 
(1) We have MCÇ(B)'CC =Q<Aça(B)' , i.e.. n(MA(B)') = AIB t. 
(2) I4 bave AÇ(B)' C = (M' C) V (Aa(B)'), i.e., A2B' = 
(A2oA') V (Al 
I 
(3) Let B1 :: JwB Jw be the ba$ic extenion of B ç A. and set 1 
Then H(R(M  AÇ(B)'))= B (A) '. In particular.  



GALOIS GROUPS 2827 

Pro@ (1) This follows from Lemma 6.1 and the identitv Ga<A f-/(C ® Aa) ' = 
bi®C. 
(2) This is due to the fact that Nc,(B) = Iii ç? (Iii çl Nc,(B)')'. 
(3) This follows frolll Part (1) and (6.3). [] 

Pro@ For w, choose a faithfifl normal state w0 on .4 a and put w := w0 o Ta. By [7, 
Theorem 3.9], there exists a (uuique) conditional expectation EB from A onto B. 
Let eB 1)e the Jones projection of B: eBA(a) = A(EB(a)). So BI = (AID {eB})'. 
Since eB G B1 ç (A(') ', we find that B1 = A V (B ç (A()'). From Lelmna 6.2 (3), 
it follows that / = c(A) V Ï?(Ï ç/N(B)') ® C. From Lemma 4.6, we find that 

I EFERENCES 

[1] H. Araki, R. Haag, D. Kastler and M. Takesaki. Extension of KMS states and chemical 
potential, Commun. Math. Phys., 53 (1977), 97-134. MR 56:2042 
[2] 3. De Cannière, On the intrinsic group of a Kac atgebra, Proc. London Math. Soc., 40 (1980), 
1 20. MR 83d:46079 
[3] J. Dixmier and O. lvlaréchal, Vecteurs totalisateurs d'une algèbre de von Neumann, Commun. 
Math. Phys., 22 (1971), 44-50. MR 45:5767 
[4] M. Enock, Sous-facteurs internédiaires et groupes quantiques mesurés, J. Operator Theorv, 
42 (1999). 305-330. MR 2000i:-16057 
[5] M. Enock and J.-1M. Schwartz, Systèmes dynamiques génératisés et correspondances, J. Op- 
erator Theory, 11 (1984), 273-303. MR 86g:46097 
[6] M. Enock and J.-M. Schwartz, Kac Atgebras and Duality of Locatty Compact Groups. 
Springer-Verlag, Berlin (1992). MR 94e:46001 
[7] M. Izumi, R. Longo and S. Popa, A Galois correspondence for compact groups of automor- 
phisms of von Neumann algebras with a generalization to Kac algebras, J. Funct. Anal.. 155 
(1998), 25-63. MR 2000c:46117 
[8] H. Kosaki, Extension of Jones' theory on index to arbitrary factors, J. Funct. Anal., 66 
(1986), 123-140. MR 87g:46093 
[9] J. Kustermans and S. Vaes, Locally compact quantum groups, Ann. Sel. École Norm. Sup., 
4 e série, t. 33 (2000), 837-934. MR 2002f:46108 
[10] J. Kustermans and S. Vaes, Locally compact quantum groups in the von Neumann atgebra 
setting, To appear in Math. Stand.. 
[11] M. Pimsner and S. Popa, Entropy and index for subfactors, Ann. Sel. École Norm. Sup., 4 e 
série, t. 19 (1986), 57-106. MR 87m:46120 
[12] J.-M. Schwartz, Sur la structure des algèbres de Kac, I. J. Funct. Anal., 34 (1979), 370-406. 
MR 83a:46072a 
[13] . Str,til,, Modular theory in operator algebras, Abacus Press, Tunbridge W'ells, 1981. MR 
85g:46072 
[14] M. Takesaki, The structure of a von Neumann algebra with a holnogeneous periodic state, 
Acta Math., 131 (1973), 79-121. MR 55:11067 
[15] T. Teruya and Y. Watatani, Lattices of intermediate subfactors for type Ill factors, Arch. 
Math.. 68 (1997), 454 463. MR 98d:46071 
[16] S. Vaes, The unitary implementation of a locally compact quantum group action, J. Funct. 
Anal., 180 (2001), 426-480. MR 2002a:46100 
[17] T. Ymnanouchi, On dominancy of minimal actions of compact Kac algebras and certain 
automorphisms in Aut(.A/.A), Math. Stand., 84 (1999), 297-319. MR 2000g:-16101 
[18] T. Yamanouchi, The Connes spectrum for actions of compact Kac algebras and factoriality 
of their crossed products, Hokkaido IVlath. J., 28 (1999), 409-434. MR 2000g:46100 



2828 TAKEHIKO YAMANOUCHI 

[19] T. hamanouchi, Uniqueness of Haar measures for a quasi \¥oronowicz aIgebra, Hokkaido 
Math. J., 30 (2001), 105-112. Ml 2002e:46075 
[20] T. Yamanouchi. Description of the mltomorphism group Aut(,4/,4 c*) for a minimal action of 
a compact Kac algebra and its application, .]. Funct. Anal., 194 (2002), 1-16. 

DEPARTMENT OF NIATHEMATICS. FACULTY OF SCIENCE, HOkkAIDO UNIVERSITY. SAPPORO 060- 
11810 JAPAN 
E-mail address: yamanuc@math, sci. hkuda±, ac. jp 



TRANSACTIONS OF THE 
AMEIRICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7. Pages 2829-2855 
S 0002-9947(03)03273-2 
Article electronically publiqhed on March 14. 2003 

COMPOSITION OPERATORS ACTING ON 
HOLOMORPHIC SOBOLEV SPACES 

BOO IRIM CH()E, HYUNGWOON KOO. AND WAYNE SMITH 

ABSTRACT. \Ve study thc action of composition operators Oil Sobolev spaces 
of analytic flmctions having fractional derivatives in some weighted Bergman 
space or Hardy space on the unit disk. Criteria for when such operators are 
bounded or compact are given. In particular, we find the precise range of or- 
ders of fractional derivatives for which all composition operators are bounded 
on such spaces. Sharp results about boundedness and compactness of a com- 
position operator are also given when the inducing inap is polygonal. 

1. INTF¢ODUCTION AND STATEMENT OF IESI'LTS 
Let D be the unit disk in thc complcx plane.  shall write H(D) for the «lass 
of ail holomorphic functions on D. Let .s k 0 be a l'eal muni»er. Following [BB], we 
defilm the flactiolml del'ivative tbr J'G H(D) of ordcr  bv 
f(z) = (1 + n)az , z  D 
where 0 z is the Tavlor series of f. 
In this pal)er, we are going to investigate composition operators a«ting on holo- 
moI'phic Sobolev spaces defined in terlns of fractional derivatives. To introduce 
those hololnorphic Sobolev spa«es, let us first recall SOlne wcll-klK»Wn flmction 
spaces. For 0 < p <  and a > -1 the weighted Bergman space 4 v is the 
space of all f  H(D) for which 
= f I/(z)lp( -Izl)  dA(z) < 
,If,, 
where dA is area measure on D. Also, the Hardy space H p is the spa«e of all 
g  H(D) for which 

]o " dO 
O<r<l 

We will oit.Pli use the following notation to allow unified statements: 
AP_I = H p. 
TMs notation is justified bv the weak-star convergence of (o. + 1 )(1- [z[) a dA(z)/zr 
to dO/2zr as a -- -1. 

tt.eceived by the editors April 4, 2002 and, in revised form, August 6. 2002. 
2000 Mathematics Subject Classification. Primary 47B33: Secondary 30D55, 46E15. 
Key words and phrases. Composition operator, fractional derivative, Bergman space. 
The second author's research was partially supported by KF/F2001-041-D00012. 

()2003 American iNIathematical Societ 



2830 B. CHOE, H. IxOO, AND W T. S/vIITH 

Now, for p > 0, s OE 0 and et >_ -1, the hololnorphic Sobolev spa.ce A, s is 
defined to be the space of all f  H(D) for which *f  APo.  will often write 
H = A   define the norm of f  .4  by 
[[fIIAL = 
Of course, we are abusing the terre "norm", since I1 IIAe, does llot satisçv the 
t.rialagle inequality for 0 < p < 1. but in this ce (f, 9) IIf - 
 9llA. defines 
a translation-invarian metric on .4 which turns Ag into a complete topological 
vector spae. 
A fraction   H(D) that satisfies ç(D) C D induces the composition operator 
C, defined on H(D) by 
Throughout fifis paper the symbol ç will always represent a holomorphic self-lnap 
of D. In this paper we study the action of composition operators on holomorphic 
Sobolev spaces. This setting allows a unified trea.tment of composition operators 
_ 
on Hm'dy spa.ces (H p = .4 p 1.0), weighted Bergman spaces (.4 = P -1), 
and Dirichlet-type spaces 
(A,), where extensive resem'ch has already been done. 
The book [CM] is a good introduction to this work. The main results in this paper 
may be viewed as smmnarizing well-known boundedness and compactness results 
for composition operators on these spaces, and then extending them to the Sobolev 
setting. 
It is a well-known consequence of Littlewood's Subordination Principle that everv 
composition operator is bounded on A for every p > 0 and   -1: see [MS]. It is 
natural to ask how this extends to the spaces A. when s > 0. For p > 0, j > -1, 
sj  0 (j = 1.2) with a - 2 = p(s - s2), we have the following equivalence (see 
Theorem g.12 in [BB]): 
(1.1) P  A p 
AI -Sl 2,82 " 
That is, these spaces are isomorphic and have equivalent norms. In particular, when 
s < «+ we hae A s  Aa_sp. Thus it follows that eve- composition operator is 
bounded on A s when s < +t. The general situation is described in the following 
, p 
theorem. Just the statement of this and out other main results are given in this 
section. The proofs will corne later. 
Theoreln 1.1. Let p > O. s  O and   -1. 
+ ten ever composition operator s bonded on A, s. 
(a) If .s < , 
(b) Ifs- + 
p 
(i) p  2 or  = --I, then eve composition operator s bounded on A, s. 
(il) p < 2 and  > -1. ten some composition operators are hot bounded 
o 
(c) If s > + then some composition operators are hot bouded on A 
The case  = -I in part (b) corresponds to s = 0, and  previously mention 
everv, composition operator is bounded on H  = 4P-,0- The case o = -1 in part 
(c) shows that this does hot extend to H for a range of positive s, 
for the Bergnmn-Sobolev spaces. 
The boulldS on s in Theorem 1.1 can be extended when the inducing map of the 
composition operator is univalent or, more generally, of bounded valence. 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2831 

The upper bound 
given in §6. We do 
sharp, but another 
be extended to the 
The equivalence 
for oel > -1 with o1 
inclusion relations: 
(1.2) 
(1.3) 

Theorem 1.2. Let p > O, s > 0 a,d a > -1. Assume that q is of bonded valence. 
(a) If p>2 and s < a+ then C is bounded on A . 
a+l 1 then C is bouttded on A,. 
(b) If p<2 ands< +, 
s < a+ inpart. (a) forp > 9 issharp: auexamplewillbe 
a+l 
hot know whethcr the ui)per bouud s <  +  in part (b) is 
examl)le will bc given that shows that the upper bound cmmot 
1)omd «+ from part (a). 
p 
(1.1) does not extend to the limiting case œee = -1. However, 
+ 1 = p(st - sœee), we have the folh)wing Littlcwood-Paley-type 

p < 2 === 4 p C AP_I,se, 
-- - (3i , I 
,P 
p> 2 === . -1,s2 C 
__ - Ol,S I • 

hlclusion relations for different values of p are also kllown. 
aj > -1, si > 0 (j = 1,2) with +2 +œee = st - sœee, we have 
  P P2 

For 0 < Pi < P2, 

Let. p > 0, c >_ -1. and s >_ 0. Note that ,are have .4, C A (+2)/(+2-p) for 
ps < c + 2. as a special case of the above inclusion (s2 = 0, ni = (2)- hl case 
ps > c + 2, inclusion relations with other types of fimction spaces are known a.s 
follows: 
(1.5) 0 < ps - (( + 2) < p == -4Pa,s C As-(a+2)/p, 
(1.6) ps=c+2  4 p cVMOA. 
Here, Ae denotes thc holomorphic Lipschitz space of order e, 0 < e < 1, and 
VMOA denotes the space of holomorphic functions of vanishing meml oscillation. 
The definitions and more information on these spaces can be round in [CM] for A 
and [G] for VMOA. For details of ail the inclusions mentioned above, see Theorenl 
5.12, Theorem 5.13, and Theorenl 5.14 in [BB]. 
The boundedness (conlpactness) of a composition operator on a smaller space 
often implies the boundedness (compactness) of the operator on larger spaces. This 
general philosophy and the inclusion relations nlentioned above lead to natural con- 
jectures. The nlethods developed below in §2 to addrcss tllese conjectures require 
some restriction on the parmneters. In particular, the case (1.2) is left open since 
out nethods do not apply when the target space is a Hardy-Sobolev spa.ce. 
Theorem 1.3. Let X C I" be any of the inclusion relation in (1.3) - (1.5), and 
assume for iclusion (1.3) that sœee < 1. a, for mchtsio,(1.4) that c2 > -1 and 
s2< 1+(1+c2)/P2. 
(a) If C, • X --, X is bounded, the Cv • }" --, Y is bounded. 
(b) If C " X  X is compact, then C " Y - Y is compact. 
Inclusio,1 (1.6) was left out of the preceding tlleo,'eln, but we have the following 
partial result in that case. 
Theorem 1.4. Let p > O, o >_ -1. s > 0 and assume ps = a + 2. 
(a) If C, " A, ---, AP, is bounded, then C, • VMOA -- VMOA is bounded. 



2832 13. CHOE, H. KOO, AND "V. SMITH 

VIIOA is compact. 

\Ve also mention the elementary inclusion relations that. for ail p > 0, s >_ 0. 
c _> -1, and e _> 0, 

,tP P 

In §3 we will give a result analogous to Theolem 1.3 for these inclusions, but with 
SOllle restrictions on the parameters. 
As a first application of Theoreln 1.3. notice that it can be used to prove the 
case c > -1, p > 2. and s = +__! in Theorem 1.1. Then H p CAPa s bv (1.3), and 
so everv COlnposition operator is bomded on AP, bv Theorem 1.3. hl the other 
direction, once criteria for Co to be bOulded or compact on the larger spaces are 
known. Theoreln 1.3 can be used to l)rovide necessary conditions fol bouldedlleSS 
or COlnpactness of Co on the slnaller spaces. For example, bv taking Ae as the 
larger space, we bave the flllowing consequence, which has |)een kllown for p _> 2 
(Theorem 4.13 in [CI[]), while it bas been known to be false for p = 1 (p. 193 in 
[Chi]). So. the gap l < p < 2 is now filled in. A more general version is proved as 
Theorem 3.3 below. 

Theorem 1.5. Let p > 1 aTtd suppos« C " II' -- Hï is bo«vded. Then the angular 
derivative of ç exists at all points ç  i)D whcre the radial limit (Ç) of ç exists 

A basic problem in the studv of conlposition operators is to relate function- 
theoretic properties of ç to operator-theoretic properties of the restriction of C'o to 
• 4 p bv 
various spaces, as in Theoreln 1.5. When ps < a + 1. we have .4Pa s "  a-sp , 
(1.1), and criteria for (7 • 4 pi  4 p= to be bounded or COlnpact are klloWll. The 
 O1  2 
characterization is that a generalized Nevanlinna colnting function for , satisfies a 
growth condition if p2 >_ pi, or an integrability condition if p2 < p: sec [Sml] and 
[SY]. The results in [Sml] and [SY] do hOt apply when ps >_ a + 1 in either the 
domain or the target space. |11 that case, criteria in the form of Carleson lneasure 
conditions for a measure defined using a lnodified comt.ing fimct Joli call be obt ained 
as in Theorem 2.6 below, with sonle restrictions on the paralneters ctj. pj, and si. 
This Carleson-type criteria in Theoreln 2.6 will be used to prove Theorelns 1.2 and 
1.3. \Ve also lnention that for the special case p = 2 other teclmiques m'e available. 
since the llOrlll of a ftlllctioll in A, lnaV be given in terlns of its power series 
coeflïcients. These spa.ces are examples of what are called weighted Hardv spaces 
in [CM], which is a good reference for colnpositiol operators acting on these spaces. 
Characterizing when a colnposition opera.tor is bounded on HsP, s > 0, seelns 
much harder. The diflïicultv is that (1.1) does not provide ail isonlorphisnl with 
a space of functions defined with full derivatives, and the llethods used to prove 
Theorem 2.6 do hot apply. We have froln Theoreln 1.1 that. for anv p > 0 and 
s > 0. there exists a functiol ç such that C is hot bounded on Hs p. A positive 
result is that C is COlnpact on certain Hs p whelever ç is of bounded valence and 
ç(D) is COlltained in a polygonal region COlltained in D. This is the special case 
pi = P2 of the following result. Fol" a polygon P inscribed in the trait circle, let 
O(P) denote 1/rr times the measure of the largest vertex angle of P. 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2833 

Theoreln 1.6. Let 1»2 _> p, > 0 and as.saine 0 <_ s < min{&p2, }'1 Let ç be a 
h, olomorphic flo.**ctio' of bo'uded vo, lence tat:ig D ito a polygon P iscribed i 
th u, it circle. IfO(P) < p(1--sp2) thf @ " Hf   Hy 2 is çom]ioct. 
p2 (1- sp ) " 
When s = 0 and p = P2, this has long been known: see [ST]. When s = 0 and 
p  p, this result is basicallv contained in [Sml]. These results (when s = 0) do 
hot require the hypothesis of bounded valence.  will prove a more general result 
in Theorem 5.5. 
In the next section we develop the change of variable methods that we use to 
study composition operators, which we then use to give Carleson measure-type 
criteria for these operators to bc bounded or COmlmct. These criteria are then used 
in 3 to prove Theorem 1.3. Next. in 4. the proofs of Theorem 1.1 and Theorem 1.2 
e given. Simple geometric criteria are then developed in 5 for loundedness and 
compactness of a composition operator between holomorphic Sobolev spaces when 
the inducing map is polygonal. The pal»er concludes, in 6. with several examples 
which denlonstrate that out theorems are sharp. 

2. ]A('KGttOUND: CARLESON-TSt'E CRITERIA 
Oto" al)proach t.o studying COlnposition operators on the spaces A, involves a 
change of variable ri'oto z to w = ç(z). The equivalen«e (1.1) allows us to assume 
that s is an integer, and then standard non-retiraient change of variable methods 
can be applied. This gets quite compli«ated when s is an integer greater than 1. 
Thus, for simplicity and claritv of presentation, we confine our attention to the case 
,s = 1. Thisenables us to cover parmnetersp.  and s with +(1-s)p > -1 
by using the equivalence A,  A2+(l_)p.1 ri'oto (1.1). The change of variable 
method for s = 1 is summarized as follows. 
For a hololnorphic lnap  • D  D and w G D. define the modified counting 
function Ç,(ç, w)corresponding to the measure (1- [z[)dA(z) by 
oEç,(ç, w) =  Iç'(z)lP-(1 -Izl)  
where the sure is over the set {z • ç(z) = w}. As usual, the zeros ofç- w are 
repeated according to their multiplicity. The change of variable formula we need 
ses the lllestlre 
Then. bv the area formula (see Theorem 2.32 in [CM]), we have the following «halage 
of variable fornmla. 

Proposition 2.1. Let p > 0 atd c > -1. Then, we bave 

[(f ° ç)'(z)[P( 1 --Izl2)dA(z) = / [f'(w)[Pd#, (w) 
• . (D) 

for functions f  H(D). 

Note that Proposition 2.1 cannot be directly applied to the case s = 1, because 
Ttf(z) = f(z)+zf'(z) by our definition. This difficultv is overcome bv the following 
proposition. 'e will off.en write X < }" if X _< C" for some positive constant C 
dependent only on allowed parmneters, and X  }" if X < }- < X. 



2834 B. CHOE, H. KOO, AND W. SMITH 

Proposition 2.2. Let p > O. c >_ -I and a E D. 
ri. we bave 

for f e H(D). 

Then, for every positive integer 

n--1 

Pro@ We prove the proposition for a = 0. The proof for general a is sinfilar. The 
eq**ivalece IIIIA.  2=0 Ilfl)llA is proved in Thorem 5.3 of [BB]. Thus, 
llfl A > n--1 
--0 If()(°)l ÷ IIf()llAa is clear by subharmonicity. 
Now, we prove the other direction of the inequalities. Since H(D) is dense in ail 
holomorphic Sobolev spaces bv Lemma 5.2 of [BB], it is suflîcient to show that 
(2.1) IIfllA < If(0)l ÷ IIf'llA, f  H(D). 
First, assume either ve > -1 or 0 < p _< 1. Let. f  H(D). For each/3 > -1, we 
bave bv Theorem 1.9 of [BD], 
1 /D Tf(w)Gl(z)(1 -Iwl=)aA(u') 
f ( ) : - . 

where 

1{1 
Ge(Z)=z (1-z)+ 1 . 
Therefore, choosing ./3 > -1 sufficiently large, we have by Lenmm 4.1 of [BB] 
(ve > -1 or 0 < p _< 1 is used here). 
(2.2) IIfllA < IIfllA+  IIfllAX+ ÷ IIf'll o+ 
A p . 
It is easy to see tlmt, given g > 0, there exist a constant C > 0 and a COlnpact 
subset Iç= {z  Dlz ] _< r< 1} ofD su«h that 
[If[l£+ < ellfllA ÷ C sup If(z)l. 
Taking  > 0 sufiîciently small, we have by (2.2), 
IlfllA¢ £ IIf'llA+ ÷ sup If()l 
< If(0)l + IIf'llA + sup If(z) -- f(0)l 
< If(0)l ÷ IIf'llA¢ ÷ sup If'(z)l. 
Since supe/,- If'(z)l £ IIf'llAg by the subharmonicity of If'l p, we obtain (2.1) as 
desired. 
Now, consider the case  = -1 and p > 1. Note that 
[f(e °) - f(0)l < If'(te°)ldt. 
Therefore, by Minkowski's inequality, we bave 
[If- f(0)[[H <  [f'(te°)[PdO dt <_ IIf'llH». 
which implies (2.1). The proof is complete. [] 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2835 

Having seen Proposition 2.1 and Proposition 2.2, it is now clear that the behavior 
of C, when the target space is A p depends on that of the measure ttp, For 
botm(ledness and compactness of C, the criteria for It, turn out to be Carleson- 
type conditions in certain cases. To pi'ove it, we ueed a couple of lemnms. 

Lemma 2.3. A bounded subset of any of the spoces APo, s. 
a >_ - 1, is a normal fam ily. 

where p > O. s >_ O. and 

Pro@ First assume o > --1. Using (1.1), a bounded set X in AP.s is also bounded 
in some A p where  is a nonnegative integer. Recall that there is a constant 
such that 

I.q(,*')l 
for ail g • A (see, fol" example, Tlmoreln 7.2.5 in [R1]). By Prol)osition 2.2, this 
shows that the functioIs in X are mfiforlnly bomMed on COlnpact snt)sets of D. 
Hence X is a normal family. The proof of the resull fi)l'n = --1 is sinfilar, since 
.4ç,, æ c .4ûy æ by (1.4). The l,roof is COlnplete. 

In the next lelmna we will need the estimale that if a > -1 and/3 > 0, then 

fD (1--11)" dA()  1 
(2.3) Il - zl ++ 

A reference is Theorem 1.7 of [HKZ]. 

Lemma 2.4. Let p > O. c > -1 and s > O. Let N > +- - s. Put g(z) = 
(1 - zO) -v for a, z  D. Then. we have 
IlgallAs,.  (1 -I1)--+,  • D 
where the constants in this estimate depend on N, s. a. and p, but are mdependent 
ofa. 

Pro@ First, assume c > -1. Let k be the smallest integer satisfying k _> s. Then, 
we have AP,  APa+(k_s)p.k by (1.1). Thus, by Proposition 2.2, we bave 
- { fD (l _ D[)'+(k-)PA(z) } /P [ça AP "" 1 + Z cN'j[a[J + CN'k]a[k 11 - zg[P(N+k) 
where c,j = N(N + 1)...(N + j- 1). Thus, by (2.3), we have 

]g AP  

where C = C(N,s,c.p). The desired estinmte follows. Next, assume c = -1. 
API A p 2p 
Note that o,a/p+s C _, C Ao.  by (1.4) and thus 
Ilgoll A,:,..," £ IlgoIIA,.., £ Ilgoll,«/=,:,.:,/,+., • 
O11 the other hand, we have 

II.qll.g.,,.,  ( -I,1) --÷  
by what we have just proved for the case ct > -1. This completes the proof. 



2836 B. CHOE, H. KOO, AND W. SMITH 

For anv arc I C BD define the Carleson square over I tobe 
SI = {re ie • 1 -II] < r< 1. e e  I}, 
where [1[ is 1/(2rr) rimes the Euclidean length of I. Also, let ô denote the complex 
differential operator, i.e., ôf = f' for f  H(D). 
The next lemma asserts that certain operators are compact.  review the 
definition, since when p < 1 the spaces involved are hot Banach spaces. Suppose 
X and }" are complete topological vectors spaces whose topologies are induced bv 
metrics. A contilmous linear operator T " X  }" is said to be compact if the image 
of everv bounded set in X is relatively colnpact in } Due to the metric topology of 
}" T will be compact if and onlv if the image of every bounded sequence in X bas 
a subsequence that converges in }'. Also, linearity of T allows us to only consider 
sequences in the mfit ball of X. 
In the following lemma, part (a) is well known: see Theorems 2.2 and 3.1 in IL1]. 
Part (b) is certainly known to experts. For example, the case k = 0, p = q, and 
a > -1 ocçurs as Theorem 4.3 in [MS]. A proof is included here since we do hot 
know a reference. In out application, we will take k  1. 
Lemma 2.5. Assume that one of the followin9 three conditions holds: 
(i) a>-l.O<pGq; (ii) a=-l,p=qk2: (iii)«=-l.0<p<q. 
Let k be a nonne9ative ite9er and p be a positive finite Borel measure on D. 
(a) ô " " A  Lq(d) is bounded if and only if 
O (]IIq+q(«+)/v) . I C 0D. 
I+(SI) 
(b) 0  . A  Lq(dp) i+ complet « nd ol if 
c+.> C+> = + 
O. 
Moover. the norm q the nap in (a) satisfies the inequality 11011 +  CIlll, 
IIl  tSe supremum of the quanlity <S0/llq+q< +2v o,,e I c OD. 
Proof.  provide a proof of {b). ç first prove the suciellCV. So, assunle 
and let {f.. be a bounded sequcnce in A, say of norm at lnost 1/2. We must 
show that {f} contains a subsequence whose k-th derivatives converge in Lq(d}. 
ecall that we have observed that a bounded set in A is a normal falnily, and 
so bv subtracting the limit function and re-indexing an appropriate subsequence, 
We may assaille that 11511A¢  1 and that {f} and hence {f t converges to 0 
uniformly on compact subsets of D. We need to show that {f[)} converges to 0 
in Lq(dp). Let e > 0 and write 
liN& IIL.(d.) = Ifa)[qdp + Ifa)[qdp, 0 < r < 1. 
The first tenn is easilv handled. For any fixed r  (0.1), the uniform convergence 
of {f)} to 0 on rD allows us to find N(r) such that 
< 
D 
Turning to the second terre, by hypothesis we can choose r = r  (0, 1) so that 
thc mesure dr(w) = XDD(W)dp(w) satisfics v(SI)  zI] , whenever [I]  1- r. 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2837 

where fl =/,'q + q(2 + o)/p. For II1 > 1 - r, we subdivide I into m arcs of length at 
lnost l-r, where 'm 5 [I[/(1-r)+ 1  2Il[/(i-r), and observe that SI(DrD) 
is contained in the Carleson squares associated with the smaller arcs. Thus. the 
previous estimate shows that 
oe(S/)  e(1 - r)   2ellI  
in this cse as well. Note that we used  2 1, which is a consequence of the 
hypotheses, for the last inequality. Thus, xve sec rioin (a) that there is a constant 
Ç such that 
Slll) 
Combilmd with the previous estimate, this shows that IIf} ) I](d,)  0 as required. 
Now, wc prove the necessity. Suppose (2.4) is false. Tlmn there exist a constant 
C2 > 0 and a sequcnce of arcs I, C 0D such that I, I  {1 and 
(2.5) (SI,,)  clI,l +(+/. 
Let  = lIl and ( Ç OD be the center of I, fi»r each . Fix a large integer 
N > (+ 2)/p. Let g,(z) = (1 - (1 - )zÇ,) -N and put f, = gllgA. - Note 
that [10,,llg  a;Np++z by Lemma 2.4. Tlms, {f} converges uniformlv to 0 on 
compact subsets of D. Now, using the COlnpactness of 0 k • A  Lq(dp), pick a 
subsequence of {f,, } whose k-th derivatives converge to 0 in Lq(dl t) and use the 
saine notation for that subsequence. Note that l1 - (1 - a)zÇ]   for z e SI,, 
and n large. Thus, by (2.5), we bave 
(a)  
for ail large z. This is a contradiction, because IIf, I1)  0. The proof is 
complete. 
Now, a change of variables md standard m'gumelts give us the following Car- 
leson measure characterizations of 1)oundedness and compactness. As discussed in 
the first paragraph of ihis section, we restrict out considera(ion of the orders of 
differentia¢ion to certain rmlges: mmlvsis of the general case seems too complicated 
for this paper.  also lnention again that when 8p ( ç 1 or p  2, other methods 
are available and much more is known: see the discussion following Theorem 1.5 in 
the Introduction. 

Theorem 2.6. Assume ttat one of the Jbllowing three conditions holds: 
(i) O' 1 > -1.0 < Pl _ P2: 
(il) a = -1,pl = Pz >_ 2; 
(iii) cri = -1.0 < Pi < 
Also. as.sume cz >-1 ad 

o 1 ÷ 2 1 a2 + 1 
(2.6) 0_<s < 1+ , 0_<s <l+-- 
Pi P p 



2838 13. CHOE, H. KOO. AND W. SMITH 

• A v is bounded if and on lg if 
(a) c A2,.,--, , 
(_.,) p,+(l_)p2(SI) : 0 [I[ (2+)p2/p1+(1-s1)p2 • I C 
(b) C • A p  A p is compact if and only if 
181 282 
P p2,a2+(1-s2)P2 " 
Pro@ Here, for brevity, we prove the suciency for boundedness and the necessitv 
for compactness. The other ilnplications can 1)e seen 1)y ey lnodifications. Also, 
let p  for silnl)licity. 
: Pp2,a2+(1-s2)p2 
First. we provc the siCiCllCy for bomdedness. One may easily modify the proof 
for conq)actness. So, suppose that p satisfies (2.7). 
Note that 
1)y the first part of (2.6). Thus, by Lelnma 2.5 (a) (k : 0), we have 
for fimctions g holomorphic on D. 
Also. note that a2 + (1 - s)p2 > -1 1)y th(' second part of (2.6). It follows flore 
(I.I), Proposition 2.2. Proposition 2.1 and (2.9) that 
P2 
5 [f(ç(0))l + IIf'll_+(+.l»«,)_. . 
Now, bv Proposition 2.2 and (1.1) again, we see that the sure in the last line above 
is equivalent to 

" P2-- +( al)P2/Pl 
Next, it is clear that [R s' f(0)[  IIfl[.4,Sl- Also, it is easy to verify using Lemma 
2.5 (a) (k = 1) that 
< I I/IIA«EE = II/IIA:,I 
II0 '/IIAX_+,+I»» '  
Putting these estinlates together, we conclude the boundedness of C " 4 pi 
- 1,81 
 2,2 " 
A p is compact. Suppose that (2.8) does not hold. Then there exist a constant 
C > 0 and a seqllenCe of arcs In C OD Sllc that IIl  0 and 
(S[n) OE Cin[ (2+eI)p2/pI+(1-sI)p2 
Let  = ILI and Ç  0D be the center of I for each n. Fixa large integer X > 
(a+2)/p s. Lette(z) (1 (1 )zç) - and put f 
..... IiIIAx.I" Note 
[[ff[IA,l   -81+(2+1)/1 by LIIII OE,. TlllI, {A } COllVFge unifoFllll}" tO 
0 on colnpt subsets of D. Therefore, using the colnpactness of 
4 p2 and 
APa,s=, we mav. pick a subsequence of {f o ç} that converges to 0 in . . 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2839 

use the saine notation for that subsequence. Now, first using Proposition 2.1 and 
then proceeding as in the proof of Lenlnla 2.5, we have 
IIf ° llAo,s  IIf  I[A= 
2+(l--s2)P2,1 
 5 -(+«)/'. Il-(1 
>C 
for all large n. This is a contradiction, because IIf o IIA=.  0. The proof is 
complete. 

3. FROM SMALL SPACES TO LARGER SPACES 
%'e now turn t.o the proof of Theorem 1.3. For convenien«e we divide the theorem 
into more easily managed pieces, considering ea«h implication separately as well 
boundedness and conlpactness. 
Theorem 3.1. Let pi, si and j (j = 1,2) be as in tbe bypotbeses of Theorem 
2.6. In addition, assume tbat  -  = si - s2. 
Pl P2 
(a) C A m v (compact. resp.) if and onl'y if C is 
• ,  A., is bounded 
bounded (compact, resp.) on A 
2,8 2 • 
(b) If C9 is bounded (compact, resp.) on 
Proof. Note that (2 + a)p2/pt + (1 - 81)P2 = 2 + 2 + (l -- 82)P2- Thus, (a) follows 
v= by (1.4). Thus, (b) follows from 
from Theorem 2.6. Also, noie that 4 ma,s C Aa=,s = 
(a). 
It is straightforward to check that when a = -1 and pi = P2  2. the hypotheses 
(2.6) in Theorem 2.6 are equivalent to s2 < 1 in (1.3). Thus, Theorem 1.3 with 
inclusion (1.3) is an immediate consequcnce of Theorem 3.1. Similarly, when 
-1 and 0 < p < p2, the hypotheses (2.6) in Theorem 2.6 are equivalent to a2 > -1 
and s2 < 1 + (1 + a2)/p2 in (1.4), and so Theorem 1.3 with inclusion (1.4) follows. 
The proof of the next theorem uses some properties of the pseudo-hyperbolic 
distance p on D. Recall that the pseudo-hyperbolic distance between points a and 
b in D is given by 

a -- b 
p(,b)= ï-2 I 

We use D(a, r) to denote the pseudo-hyperbolic disk of radius r and center o. Recall 
also the well-known and useful identity 

- b 12 (1 -112)(1 -Ibl 2) 

a,bD. 

In particular, it is a consequence of this that 
(3.1) I 1 - bl  1 -I12  1 - I12 
whenever b  D(a, 1/2). 
The next result covers the inclusion (1.5) in Theorem 1.3. and so completes its 
proof. 



2840 B. CHOE. H. KOO, AND W. SMITH 

Theorem 3.2. Let p > O, o > -1 and a+_A2 < s < 1+ a+œe If ('o is bounded 
-- p P 
(compact, resp.) on 
Pro@  first prove the assertion on boundcdness with the additional assumpt ions 
that o > -1, p > 1 and «+ < s < 1+ «+ Note that a+(1-s)p> -1 and 
P P 
a p bv (1.1). Choose a  D such that ]ç(a)[ > 1/2, and 
therefore A, m . a+(1--s)p.1 - -- 
consider the test function foc(z) = log(1 -ç(a)z). Then, by Proposition 2.2. we 
have 
/ Iç(a)l Zl2)a+(1 
[IL  çll:+,l_.,.., E I - ç(a)ç(z)  Iç'(z)l( -I -'}PdA(z) 
- 
For z  D(a, 1/2), we have 1 -]z 2  1 -la z, by (3.D. Also, the Schwarz-Pick 
Lemma tells us that ç(z) 6 D(ç(a), 1/2), and so 1 -ç(a)ç(z)[  1 -Iç(a)  from 
(3.1). Using these estimates in the lt terre in the display above shows that 
(3.2) > 
For the last inequality we used that D(a, 1/2) contains a Euclidean disk with center 
a and radius comparable to 1 - ]a 2, and that ç'l p is subhannonic. 
Meanwhile, since f(0) = 0, we have 
(3.3) 
where the lt. equivalence holds by (2.3), because sp > a + 2. Putting these 
estimates together with the assumption that C " A,  A, is bounded, we get 
aDSUp 'p'(a)[ { (1-- 'ç(a)[) } -l+'-(a+)/v(l - 
This is equivalent to the bomdedness of C on A_(«+)/v; see [Mai or Theorem 
4.9 in [CM]. 
Now, consider the general case o > -1 and a+ < s < 1 + a+. Choose q > p 
 p P 
so large that q > 1 and s < 1 + a+ . Put 13 - (a+2)q 2. Then.  > a > -1 
P q P 
and +2 -- +2 Now, bv (1.4) and (1.5), we have 
q P 
4 
Also, note that + < s < 1+ + Now, suppose that C • A,» + A is 
q q • 
bounded. Then C " A,  A, is bounded by Theorem 3.1 and thus so is 
C " As-(a+2)/p  As-(a+2)/p by the result for the special case we proved first. 
This proves the assertion on boundedness. 
ç now prove the assertion on compactness. Note that As_(a+2)/p and A. 
are MSbius invariant, in the sense that every composition operator induced bv a 
conformal automorphism of the unit disk lna.ps each space into it.self, and contained 



COMPOSITION OPERATOlqS ON SOBOLEV SPACES 2841 

in the disk algebra of holomorl»hic fimctions on the unit disk that extend tobe 
continuous on the closed disk. Thus a general theorem of 3. H. Shapiro [Sh] asserts 
that compactness of C' on each of these spaçes implies that ç(D) is a relatively 
compact, suhset, of D. We recall also that 
If(0)l + ,,p{( -]:loe)t-lf'(z)l • z  D} 
is an equivalent norm on Az; sec Thcorem 4.1 in [CM]. Now, let {f} lw a l»oundcd 
sequence in A.«-(a+2)/v-  must show that some subsequence of {f,ç} converges 
in A-(+2)/v-  know tllat {./} is a normal falnily, and thus a sut)sequence 
(which we still call {f}) converges to some f G H(D) tmiformly on compact 
subsets of D. Also, if C is compa.ct on A,, then it 
4 v C A-(a+2)/p. HellCe (1 --I=l=)-+(+=)/vl'(z)l i ll, liforllllv bO, ll,ded o11 D. 
and it follows that 
I (0) -   (0)l + (1 -Izloe)l-++oe)/l () - ,f'  ()ll'()l 
uniformly on D as n  , sincc ç(D) is c'ontaincd in a compact subsct of B. This 
means that {f, o F} c«mvcrgcs to thc fim('tion 9 = .f o ç in A-(a+2)/v, and so 
C • A-(a+2)/v  A-(a+2)/v is compact. Thc 1)roof 
Criteria fi)r C to lc 1)omded or compact on A: are known. So Thcorem 3.2 can 
be used to provide necessary condit.ions for boun(ledness or compactness of Cç on 
the smallcr spaces. In particular, we recall that the bomdedncss on A: implics the 
existence of the angular derivative of ç at all points of the unit circle whcre ç has 
a. radial limit of modulus 1: ste Corollarv 4.10 in [CM]. This l)roves the following 
theorem. 
Theorem 3.3. Let 1) > O.  > -1 atd + < 
 p P • , 
is bouned, the the a9ular derivtttive of ç exists af all points Ç  OD where the 
ro.dial limit ç(ç) of ç ezist.s a,,d so.ti.sfies I(çl = 1. 
As mentioned in the introduction, the conclusion of Theorem 3.3 is false for 
 = -1, s = 1 and p = 1. Thus, for  = -1, the lower bouud 1/p for s camaot be 
decreased in general.  also give an example which shows that the lower bound 
s > a+ in Theorem 3.3 is sharp in case o > -1. See Examl)le 6.3 below. 
The proof of the next theorem is based on Theorem 2.6. So, for simt)licity, we 
restrict our consideration to the orders of differentiat.ion covered there. 
Theorem 3.4. Let p > O. a > -1. s > 0 ad 
 p 
(a) If 1 + + s > e > 0 a'nd C is bouded (compact. resp.) on A,+«. tbe 
p 
so is C on A,. 
Proq[. Let I be an arc in the unit crcle, and let ç(z) = w G SI. A standard 
argument, using the Schwarz Lemma then tells us that 1 -]z  1 -[w]  I[, and 
SO 
w  SI. Hence 
Iz,a+«+(-)v(SI)  Ille#,+(i_,)v(SI) • 



2842 ]3. CHOE, H. KOO, AND Ve. SMITH 

and statelnent (b) is now an imluediate consequence of Theorem 2.6. The proof of 
(a) is similar alld will be onfitted. [] 

We finish this section by giving the proof of Theorem 1.4 from the iltroduction. 
which we rest.ate for convenience. 

Theorem 3.5. Let p > O, o >_ -1, s >_ 0 and aasume ps = a + 2. 
(a) If C " AVa,s - A, s is bousd«d, thes C " VMOA - VMOA is bounded. 
(b) If ç is univalent and C " A, s -- 4v,s is compact, then C • VlklOA -- 
VIklOA is compact. 

III the proof below and elsewhere, we use the notation dist(a. OE) for the Eu- 
clidean distance between a point a and the bounda.ry of a set E. 

Proof. If C is bolu,ded Oll .4P,s, then ffoin (1.6) we bave that ç = Cz  4P,s C 
VlklOA. Also, it is easy to see that (7, is bomlded on VlklOA if and onlv if ç  
Vlk[OA: see, for exalnple, [Sm2]. This gives part (a). 
For the proof of (b), we recall that when ç is Ulfivalent. C is COlnpact on VMOA 
if and Olfly if 

dist (w, 0ç(D)) \(Dl(W) 
(3.4) lira = 0: 
,,,-+,- (1- [wl) 

see Theoreln 4.1 in [Sln2]. Also, it is ait easv cousequence of the Koebe distortion 
theol'em that if ç is univalent, then 

(3.5) 

( -l[2)l'(=)[  dist(ç(z),Oç(D)), z  D: 

see Corollary 1.4 iii [P]. 
First, consider the case p > 1 and c > -1. With ps = e+ 2 < p+ct + 1, case (i) 
of Theoreln 2.6 (b) tells us that C is COlnpact on AP,» if and only if/xp,p_2(SI) = 
o(1I[ ) as [I[  0. We prove part (b) by showing that this fails when C is hot 
compact on VMOA. From (3.4), if (7 is hot compact on VMOA, then there is 
an g > 0 and a seqllence {Wh} C (D) with [zt,,[ -- 1 and dist(w,,0ç(D)) > 
e(1 -Iwl). Let I,, be the arc of the unit cirçle with ceuter u,,/[w,[ and length 
[I,[ = 2(1 -[w,[). Since ç is univalent. 

L {Iç'(-')l(1 Izl2)F-2d.4(w), 
]'P'P-2(Sln) = In -- 

where w = ç0(z). Froln (3.5), Iç'(z)l(1 -Iz[ 2)  (1 -Iw,l) for w in the disk with 
center w, and radius e(1 -Iwl)/2. which yields the lower bound 

 (s) # o(llI"), III - 0. as desired. 
Hence [Lp,p_ 2 
Now, consider the general case p > 0, c _> -1 and suppose that 
is compacL With ps =  + 2, choose q as in the proof of Theorem 3.2. That is, 
choose q > p so large that q > 1 and put fl : sq- 2 > -1. Then +2 _ +2 = s, 
q p 
and so A, s C A, s by (1.4). Thus. froln Theorem 3.1 we see that C. " A, s  A, s 
is compact and thus so is C • VMOA  VMOA by the result for the special case 
that we have proved above. The proof is complete. 



(-'OMPOSITION OPERATORS ON SOBOLEV SPACES 2843 

4. COMPOSITION OPERATORS ON AP,s 
In this section we prove Theorems 1.1 and 1.2 flore the introduction. For con- 
venience, we divide these results into more easilv managed pieces. As meutioned 
in the introduction, it is well known that every composition operator is bounded 
. 4 p bv (1.1), 
on A for ail p > 0 and  > -1 Note that we have A, 
when sp <  + 1. Thus, it follows that cvery composition operator is boundcd on 
A, s whenever sp <  + 1. Thc next two theorems complete the description of the 
general situation, as statcd in Thcoron 1.1. 
Theorem 4.1. Letp>O, s>0 ad >-1. Ifs- a+l atd 
(a) p > 9 or = -1 then every composilimt operator is bonded on 4 v ; 
(b) p < 2 and a > -1. then some composition operators are uot bounded on 
Pro@ If  = -1. then s = 0 and so every composition ol)erator is b(mnded on 
A, = H v. If a > -1. p  2 and ps = a + 1. then froln (1.3) we have that H p C 
A,« Hence part (a) follows ff'oto Thcorem 1.3. since all composition operators are 
bounded on H p. 
a+l 
Turning to thc proof of (b), firsI note that A, s  Ap_l, 1 p 
P 
P is necessarv for @ to be bounded on Ap_l. 1. Thus it 
Also, ç = @z ¢ Ap_l, 1 . 
suces to show that if p < 2 there is a bounded analytic function F 
Ap_13. The 
case p = 1 of this statemeut is outlined in exercise 9(a) iii Chapter VI of [G]. That 
construction can be modified to work for p < 2. For completeness, we sketch the 
argument. 
Let p < 2 and consid«r the flmction 
f(z) =  k-llçz 2. 
Since the series for f is lacunary with square summable coecients, it is known 
that f  BMOA. This is an easy consequence of BMOA being the dual of H 1 
together with Paley's Inequality for the coccients of an H 1 flmction ([D], p. 104). 
or see [Mi] for another approach to the proof. Next, it is easy to verify that if 
:¢A,,.={wcD" 1-2 - 
then [ff(z)[  -l/P2. This leads to the approximation 
£ if,(z)lp( 1 _ izl2F«dA(z )  1 
flore which we see that f  
Av_I. 1. This is hot the l'equired example, however. 
since f is hot bounded. But since f ¢ BMOA, there are bounded flmctions Ul 
and u2 on the unit circle such that ef = Ul + fi2 where ri2 denotes the harmonic 
conjugate of u2. Here, we are using the saine notation for a boundary flmction 
and its harlnOlfiC extension. Then If'lp  IVu I p + Iv.u21 p by the Cauchy-Riemama 
uations, and it follows that there is a bounded real flmction u on the circle such 

that 
L iv,()i(1- 
Now let F = exp(u + i'ù), so that F 
Ap_,, and the 

lzl)P- dA(z) = oo. 
is a bounded analytic flmction satisfying 
proof is complete. [] 



2844 I3. CHOE, H. KOO, AND A. SMITH 

c+l 
The proof of the next theorem, covering the case s > ---, reqnires two lemmas, 
which will a.lso be used in the next section. 
Lelnnla 4.2. Let p > O. s  O. and a  -1. Then the following inclusions hold: 
(a) A,. c A ç for e > 0: 
(b)  A . for O < < p. 
Ap+a_e,l+ s C ., 
Moreover, both iwlu.s'ions are bomMed. 
We remark in I)assing lhat, when s = e = 0. a = --1 and p  2. the inclusion 
iu (a.) holds and this is just a restatement of the well-known Littlewood-Paley 
inequalily. When s = e = 0, o = -1 and 1 < p  2, the inclusion in (b) holds and 
lhis is a restalement of the (lual of the Litllewnod-Paley inequalily. 

Proof. Bv definition of the holomorphic Sobolev spaces, it is suflïcient to consider 
the case s = 0. First. consider lhe case o > -1. Then, xve have 

il lld 

.4 c 4 ' , P 
o+e Ap+a+e,l,  > 0 

4 v " A; 4 p  APo, 0 < e < p 
" p+o--e,1  o,e/p C. oe+e,e/p 
where the eqnivalences are ri'oto (1.1) and lhe inçlusions are clearlv bounded. 
Now, assmne (t = -l. Let J'G H p and put 
1 
(4.) .x(f.,.) = . If(,.d°)FdO. o <  < 1. 
Then, for anv  > 0. we have Mp(f.r) = O ((1 - r)-), which implies Mp(f',r) = 
O ((1 - r) -t-) (see Theorem 5.5 of [D]). Thus. for e > 0. integralion using polar 
coordinates shows lhat 
fb If'(z)lP(1- 'z]2)P-l+edA(z)  '(1- 
F)--(1+6)P+P-I+dF. 
With  small enough so that pO < e, this integral is convergent, and so (a) holds. 
Now, assume 0 < ¢ < p. Çonsider the case p > 1 first. Let p' be the conjugate 
exI)oncnt of p. Bv the fundamental theorem of calculus and H61der's inequality, we 
have 
•  ,{1  
[, II(e '°) - I(0)F"0  ,. II'(rd°)[& eO 
JD 

xvhere 

It follows that 

I fo I , "1 p/p' 
C = (1 - r) -p (P-a-)/Pdr < 

IIf[Iï4,, £ [f(o)l p +/D If'(:)lP(1 -- 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2845 

and the Saille is true for p = 1 by a trivial lllodification of this argunlent. This 
proves (b) fol" p _> 1 by Proposition 2.2. When 0 < p < 1, we bave tlle inclusions 
Ap/(l-e) _ Hp/(1-e) Hp 
A;_ l_e, 1 C -1,0 -- C 
wheïe the fiïst inclusion conles ri'oin (1.4). This colnpletes the proof. [] 

We next show that certain inçhlsions t)etwoen hololnOl-l»hic Sobolev spa«es are 
conlpact. 

Lemma 4.3. Let p > O. s >_ O, ¢ > 0 and c >_ -1. Then the followin 9 itclusions 
are compact: 
p 4v A v 
A,,s+e C C 
Proof. We first consider the case  > -1. Let {f} be 
4 p is compact, we nmst show that 
A, s. To show that the inclusion A,, C . 
some subsequence of {f,} converges in 4 p I1 is well known tlmt the inclusion 
P 
P is compact: ste, for examplc [Sml]. Thus there exists g 
,4 C . 
subsequenc of {f } (whic for convenienc we continue t o denote {f })suc that 
sfn  g ill .4 pt+e. Now. chooscb  (D) Sllch that h = g. It is then clear that 
4 p This completes the proof that ,4 s C 4 p is 
h Ç 4P +e,s and f  h in . ,+e,s. , 
compact. Now, since we have by (1.1), 
4 , p  Ap 
" a,s+e C Aa+pe,s+e . 
and the first inclusion is compact, we con«huh' lhe compactness of the inclusion 
A p 
Now, assulne  = -1. Choose positive nunibers 1. 2 such that 0= < a < 
min(p, pe). Then, we have the relations 
4 v  v v A v 
Al,s+ e C « p-l+pe-6,l+s+e Ap-l-6A+s Q Ap-l-6,l+s 
The first, and la.st inclusions are bounded froni Lemma 4.2. the equivalence is ffOln 
(1.1), while the relnaining inchlsion is compact hy the previously established part 
of this lemma. Hence AP -1,s+e C AP -1,s is çOlnimct. For the compactness of A p -l.s C 
A p noe that 
-l+e,s 
4 p A p  A p 4 p 
 -ls C p-l+e/2.1+s -l+e/2,s C -l+e,s" 
The first inclusion is bounded from Lemma 4.2.  the isomorphism ri'oto (1.1), 
while the renmining inclusion is compact bv a previously established palt of this 
lemlna for a > -1. Hence AP -1,s C AP -+e,s is compact. The proof is complete. 

Theorem 4.4. Let p > 0 and o 
operators are hot bounded on AP,,. 

> -1. Ifs > +1 
-- p 

then some composition 

Proof. Since a = Csoz Ç A, s is necessarv, for C to be bounded on -4P,s, it suflïces 
to show that if s > +__1, then H  \ AP, -J= 0, where H  denotes the class of ail 
p 
bounded hololnorphic filnctions on D. Suppose to the contrary that H  C APo.,. 
Then this inclusion map is continuous bv the Closed Graph Theoreln. while the 
inclusion map AP,s C Aï.s was shown to be COlnpact whclmver a < /3 in Lemma 
4.3. The hypothesized lower bound for s now allows us to choose  _> 1 such 
that c +  < .sp <_ c +  + 1. Moreover, we can choose « > 0 so Sllmll tllat 



2846 B. CHOE, H. KOO, AND W. SMITH 

P HP/(«+a+l+«-sp) from (1.4). 
0 < c + 6 + 1 + e - sp < 1, which gives Aa+a_l+e, , C 
Consequently, we bave a chain of inclusions 

tt  C APa  C A p C tt p/(a++l+e-sp) 
, a+-- 1-t-e,s " 

Thus the inc|usion H'° C H p/(a+a+l+e-sp) can be viewed as a product of a compact 
nmp and two bounded maps, and hence is compact. But {z  } is a bounded sequence 
iu H  for which no subsequence converges in H Pl(a+'++e-p). This contradicts 
the COlnpactness of the illclusion lnap, aud the proof is colnplete. [] 

The next theoreln (also stated as Theorem 1.2 in the introduction) shows that 
the upper bounds for s in Theorens 4.1 and 4.4 can be extended whell the symbol 
ç' of the COlnposition operator is of boumled valence. 

Theoreln 4.5. Let (, > -1 and let ç be of bounded valence. Assume that either 
a+l 
0_< s _< «+--v if p_>2. or'O_< s < --K- +½ if O < p < 2. Then Co zs bounded on 

Pro@ \Ve use the Carleson measure criteria frOlll Theoreln 2.6. We need to esti- 
llla[ e 
P,a+(1-s)p(qI)=i Z I'(z)]P-2(1-]z]2)«+O-S)PdA(w) 
o(z)=w 
for arbitrary arcs I C OD. 
First., consider the case p > 2. By aSSUluption we have sp _< a + 2. Note that, 
since ç is of bounded valence, there is a Ulfifornfly bounded lmmber of terlns in the 
sure inside the integra| above. Next. we set w = ç(z) and use the Schwarz-Pick 
Lemlna, which asserts that ]ç'(z)] < (1- ]w[2)/(1- [z]2), and then the elelnentary 
inequality 1 -[zl 2 <_ C(1 -[wl 2) to get that 

which from Theorem 2.6 is equivalent to boundedness of C on APa, . We uote that 
the hypothesis p _> 2 was used in getting the first inequa|ity, and sp _< c + 2 was 
used iii the second inequa|ity. 
Now, consider the case p < 2. What we have now is sp < c + 1 + -. By the area 
formula (Theorem 2.32 iu [C[]), we have 

f 
IAP,a+(1-s)P° (SI) = Jsal-' (si) Iç'(z)[P(1 -Izl2)+O->dA(z) 

Note that, since ç is of bounded valecce, we bave 

Æ I'(z)[2dA(z) = £ Z l dA(w) £ I,I 2 
-(s,} , :e_{} 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2847 

where the first ilmquality is provided by H/31der's inequality. To estilnate the integral 
above, recall that everv composition operator C is bounded on A, 13 > -1, and 
that this is equivalent to 
(4.3) __ £_1(SI)( 1 __ iz[Z)ZdA(z) 
for all I C 0D: sec section 4 in [hIS]. Silwe, by hypothesis, 2( + (1 - s)p)/(2 - p) 
> --1, we Call combine the estinmtes iii (4.2) and (4.3) to get that 
lZ,a+(l_s)p(i) 
From Theorem 2.6. this is equivalent to C being bounded on A. s. 
complete. 

The proof is 
[] 

Our final result in this section shows that the conlpactness of the inclusions in 
Lemma 4.3 extends to ail colnposition operators with a certain restriction on s. 

Theorem 4.6. Let p > O, a >_ -1 and s > O. A,ssume that either (i) o. > -1, 
0 <_ s < c+___ilp or (ii) c = -1. s = 0 or (iii)  = c+.__!p .p>_2. 
(a) (Tv A,,+« -- Ag,, is compact. 
(b) c  
• Aa_e, * -- APa,, is compact whenever ce - e  -1. 

. 4  
Pro@ We may view the action of Co - a,s+e -- APa,, as follows: 

p Ap c 
- A,. 
Aa,s+ e C a,s 
Also, if o - e > -1 then we mav view the action of C • A p  .4 s as follows: 
p C 4p 
A_«,, c A,  
The ilmlusions above are both compact by Lemma 4.3. and everv composition 
operator is bounded on A,œ by Theoreln 1.1. Thus, both operators m'e compact. 
The proof is complete. 

Carleson lneasure criteria for Co to be bounded or COlnpact Oll Hï are known; 
see Theorems 4.11 and 4.12 in [CM]. For s hot an integer, characterizing when 
Cso is bounded or compact on H seems much harder than the analogous problems 
on the Berglnan spaces. The problem is that for the Hardy spaces, (1.1) does hot 
provide isomorphisms with spaces defined using fldl derivatives. Thus. we are led 
to the following. 

Problem. Characterize ç for which C is bounded (compact) 011 HPs , 8 > O. 



2848 B. CHOE, H. KOO. AND W. SMITH 

5. (',OMPOSITION WlTH A POLYGONAL ,IAP 

Here. xve find simple criteria for the COlnpactness of the composition operators 
betv«een holomorphic Sobolev spaces induced by polygonal lnaps. 
Recall that dist(a.0E) denotes the Euclidean distance between a point a and 
the boundary of a set E. 

Lelllllla 5.1. Suppose that P is a polygon inscribed in the unit circle with a verrez 
al v and let rq(t,) be the vertex angle al v. Then. given a Riemann map  of D 
onto P. there es:.isls a neighborhood N of v such that 

I¢(z)l  ( -Iqo(z)l) -1/'(), 
( -14)  (1 -Iq(z)l) /)-I dist(v)(z). OP) 

for all z  ç-(N). 

Proof. Reca.ll that ç extends to a holneOlUorphisn of D outo P (see, for exalnple, 
Theoreln 14.19 of [R2]). Assulne v = 1 aud ç(1) = 1 for simplicity. Also, let 
q = q(v). Then, a reflection argmnent yiehis 

l l  (): C ( 1 -- Z )  + 0(I I I +',) 

for SOlllt, constant c ¢ 0 and for ail z near 1. Thus, we have 

Iv/()l  I: - l ":  I: -v)(z)l -I/  (1 -Iv)(z)l) 

for z near 1. The last equivalence in the disploEv above holds, because ç(z) is 
contained in a lontangential region with the vertex at 1. This proves the first 
equivalence of the lemlna. The second equivalence is now a consequence of the 
estimates 
(11 _]z[)]ç'(z)]_< dist(ç(z).OP)<_(1-[z])lç'(z)[, 
which hold since ç is mfivalent: see Corollary 1.4 of [PI. The proof is complete. [] 

Lemma 5.2. Let P be a pollgon inscribed in the unit circle. Assume b > -1 and 
a + b > -2. Then, there exists a constant C > 0 such that 

/oe 1-lwl) a dist(w, iï)P)bdA(w) <_ CIII 
çS I 

for all arcs I C OD. 

Pro@ Let us introduce a temporary notatiolL For an arc I C 0D with center at 
ç  0D and III = 25, we let S5(ç) = SI. 
Assmne that  is sufficiently small and P çl S5(ç) ¢ . Then. there is a constant 
C. depending only on P, such that S5(ç) c Sc5(v) for some vertex v of P. Assume 
t, = 1 for simplicity. Assuming that  is sufficientlv small so that Se, e(1) contains 
no vertex of P other than 1. note that 1 -lu, I  Il - u,] for w G PO Sc6(1). Now. 



(OlklPOSITION OPERATOIRS ON SOBOLEV SPACES 2849 

we bave 

(1 -[w])  dist(tv, OP)bdA(w) 

constant depelMing ouly on P. The estiluate for large 

[] 

Recall that D(z. 1/2) deuotes tlw pseudohyi»erl»olic disk. Let D(z) = D(z. 1/2). 
In the following we let dA,(z) = (1 -Iz]=)dA(z) for c > -1. The following lenmm 
is proved fol" a = 0 in IL2], and the saine proof works for general a. 

Lemma 5.3. Let c > -I and It be a positive finite Borel measure on D. Assume 
p > q > (I. Then, there is a constant C such that 
if and onlv if r G L(A) where r(z) = u(D()) 
A,,(D(z))" 
For a polygon P inscribed in the uuit circle, recall that O(P) denotes 1/ times 
the measure of the largest vertex angle of P. 

Proposition 5.4. Let pi > O. ai > -1. si >_ 0 (j = 1.2) and assume 
O1 3-2 1 a2+l 
(5.1) s < 1 + , s < -- 
Pi P P2 
Let  be a holomorphic function takin9 D into a pol19on P inscribed in the unit 
circle. If 
pl (O2 -[- 2 -- 82102) 
(5.2) 0(P) < 
P2(al + 2 -- SlPl)' 
• 4 1 -- A, , is bounded. 
lhen C¢ " Ol,Sl 2, 2 
Mo.reover, for functions ç of bounded valence, the second part of (5.1) can be 
replaced by the weaker condition that 
c + 2 c2 + 1 1 
(5.3) s2 _< -- if p2 >_ 2, or s2 < -- +  if 0 < P2 < 2. 
P2 P2 - 
h either case the eq,uality tan be allowed in (5.2) for P2 >_ Pl. 



2850 B. CHOE. H. KOO, AND '. SMITH 

Pro@ Let ç0 be a Rielnann nlapping of D ollto P all([ put ¢ = ç- o ç. Then 
ç = ç0 o  and Oms C¢ = Ç¢oO = Cç'C¢o. Note that CO " 4 pz  4 p 
is bomded by Throrem 1.I or Theorem 1.2. This shows that we only need to 
prove the proposition for ç = ç0- So, in the rest of the proof, we assume that ç 
is a iemann map of D onto P. For simplicity, let  = pj + j - sjpj and let 
?=2+j-jpj forj=l,2. 
First, consider the case p2 OE Pl- By Theorem 2.6, ve need to show that 
(5.4) t'p, (SI) = 0([II (2+)p/m+(1-)p) 
for ail arcs I. As in the proof of Lemnm 5.2, we onlv need to considcr I centered at 
a vcrtcx of P for which ]I] is sufficiontly small. Given such I, we havc bv Lemma 
5.1 and Lomma 5.2, 
, ]'(-i(w))]P-2( 1 -]-1 (u,)]2)OdA(u,) 
 [ dist(w,0P)u(1 -]w[)(/°-i)ud.4(w ) 
IP 
where 0 = O(P). In he last inequality we used the fact that ff2 > -1, 2  0 from 
(5.3) and tlms 
Thus, we have (0.4) by (5.2) and (5.5). Also, the smne proof works in case the 
equality holds in (5.2). 
Next, consider the case p < p.  lnav, assume ç(0) = 0. Let f  ,4 m, be 
an arbitrary fimction such that f(0) = 0. Since p < p, we have  > -1 bv the 
first part of (5.1). Thus, by (1.1) and Proposition 2.2, we lmve 
[[f][A,  [ f'(w)[m(1 --[w[)dA(w) • 
Also, we have by (1.1), Proposition 2.2 and Lemma 5.1, 
Now, define mesures 
dpa(w) = (1 -[l)dA(), 

and let 

r(z) = p2(D(z)) z • D. 
pl(D(z))" 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2851 

By Lelmna 5.3, we need to show that v E LP(p) where p = Pl/(Pl - P2). Note 
that/t2(D(z)) = 0 if z is outside of some t)olygonal region Q. ()n the other hand, 
for z G Q. we have 
p,(D(z))  (1 -Izl) »l+, 
pœee(D())  (1 -Izl2) +(/e-1)+œee', 
the first esçimate is standard and the second one çall be verified with (5.6) bv 
modiÇving the proof of Lomlna 5.2. According]y, we lmve 
(=)  ( -i=l)=-»,+,/e-,,b( ). 
[t fo]lows that r e L(pt) if and on]y if p[fl2 - fl + (1/8 - 1)72] + fit > -2, which 
urns out to be thc saine as (5.2). his completes the proof. 
Theorem 5.5. Lct pj > O. ej  -1, si  0 (j = 1,2) and assume (5.1) holds. Let 
ç b« a holomorphic function taking D into a polflgon P iscrih«d in th¢ unit circle. 
If (5.2) holds, fh«n Ç" 4 m  4 p= is compact. 
Mor«ov«r. for flmctio,s ç of bovnded valence, t« second part of (5.1) can be 
r«plac«d by the w«aker co,ditio** (5.3). 
In Examplc 6.1 below, we show that (5.3) providcs the sharp upper bound of 
+2 for s2 when p2 > 2. Whilc we do hot know whcther it docs the saine when 
P2 ( 2, the ni)per bomd for s2 when p2 ( 2 cannot be extended to :+ as is 
P2 
shown by Example 6.2. Nevcrthelcss, Examplc6.4 shows the upper bomd of 
in (5.2) is sharp in eithcr case. 
Pro@ Assume that (5.2) hokls and choosc ¢ > 0 suciently small so that (5.2) 
holds with a + e in place of  Bv Lemma 4.3 we have 4 p Ç 4 P and 
. 4 m  4P2 
the inclusion is compact. Thus, it is sufficicnt to show that C 
is bounded. In case au > -1. we sec that C¢ • A',+«,  -4Pa,s is bounded bv 
Proposition 5.4. 
So. assume 2 = -1. Note that with a2 = -1 there is no s2 satisfying the 
second part of (5.1). Thus. we only need to be concerned about the case where 
is of bomded valence and (5.3) holds. First. consider the case p2  2. In this case, 
we can view the action of @ as follows: 
Ara cv p 4p  
a+e,s  Ap2-1,se+l Ç ' -1,s 
where @ • Ama+e,s  APp-,s+ is bomded bv. ProI)osition 5.4 and the inelusion 
cornes from (1.2). Therefore, @ • A m  4 p is boCmded. Next, consider 
lE,81 " 2,8 2 
+2 _ 
çhecasep2>2. Choosepe(2,p2) ande >-1. Also, leçs=  +s2 . 
Then. we tan view the action of C  follows: 
4m c ApÇ.s  4p  
where C " A m 
+e,s  A,s; is bounded by Proposition 5.4 and the inclusion 
cornes from (1.4). The proof is complete. 
Remarks. 1. As mentioned in t.he proof above, t.here is no s2 satisfying the second 
part of (5.1) in case a2 = -1. Thus, we have no conchlsion in Theorem 5.5 for 
general ç in ce the target space is a Hardy-Sobolev space. 
2. Note that the condition (5.2) holds vacuously if a+2 +2 
P P2 



2852 B. CHOE, H. KOO, AND ,V. SMITH 

6. EXAMPLES 
rVe llOW give sevcral exmnples delnonstrating that our theorelns are sharp. For 
tiret purpose we introduce the so-called lens maps. Fol" 0 < 11 < 1 we denote by çn 
the flmction defined bv 

er(z) ' - 1 

(6.1) qa,(z) -- er(z)' + 1" z C D 
where er(z) = (1 + z)/(1 - z). Let n(D) = L. Then, çn is the Rielnann map 
of D onto the sui»set L of D bounded by arcs of circles lneeting at z = +1 at an 
angle of lift, and fixing the points -1, 0, and 1. Because of the shape of the range 
L,» such a map is called a "lens map". Note that L is contailmd in a polygon 
inscribed in the unit cir«le. 
By a straightforward calculation, we bave 
(6.'-,)  -I()1  1 - zl ', I'(z)l  I] - zl 

for z near 1. 
The first example shows that the upper bomul s < +___2 in Theorem 1.2(a) is 
 p 
slmrp. Also, this exalnple shows that thc upI)cr bound so < az+ in Theoreln 5.5 
is sharp when p2 k 2 and  is of bounded valence. 
Example 6.1. Let 1 »> 1, o >-1 and + < s < 1 + a+l Thon, thereexists a 
P P 
P 
lens map n  -4,s" In particular, C is hot bounded on Aa, s. 
Proof. Choose 0 < 1 < 1 sucientlv small so that sp k IP + a + 2 and consider 
the corresponding lens lllap ç Note that A   A P bv (1 1). Therefore, 
" ,  a+(l--s)p,1 - " 
we bave bv Proposition 2.2 and (6.2). 
 ./ I'o (z)lp(1 -Izlu)+(1-PdA(z) 
aD 

Note that (r/- 1)p + er + ( 1 - s)p <_ -2. because sp _> 'lP + er + 9. Thus, an obvious 
estinmte iii an angle with vertex at 1 shows tiret the last integral above diverges, 
as desired. [] 

1 
We do hot know whether the upper bound s < a+__! + 7 in Theoreln 1.2(b) is 
sharp. However. the next exalnple shows t.hat the upper bound cmmot be extended 
 1 
to + as in Theorem 1.2(a.). Also. this is related to the assumption, s., < =+ + , 
p  p2 
in Theorem 5.5 when P2 < 2. 
Example 6.2. For each p  [1,2), there exist  > -1, 0 < s < 1 with s < + 
 p 
and a univalent holoInorphic self-map  of D such tiret   A p In particular, 
C is hot bounded on A,,. 
Pro@ P..lones and N. Makarov have showll (see Theorem D(2) in [aMI) that for 
any p < 2, there exist a univalent holomorphic self-map çp of D and a constant 
' satisD 
c > 0 such that the integral ieans of p . 
P--I+c(2--P)   P  __ 
(6.3) (1 - ,') - I (wp- ") > 1 



COMPOSITION OPERATORS ON SOBOLEV SPACES 2853 

for some sequence 7", -- 1. Here, we are using the notation introduced in (4.1). 
Note that, for anv f E. a, /3 > -1, we bave 

a.ild so 

1 
(1 - r)+SIP(f,r)= (/3+ 1)/lç(f, r) .f (1 -t)dt 
< (/3+ 1) lllP(f,t)(1-t)13dt=o(1) 
! 
as r  1. This, together with (6.3), yields çp tg Ap_2+c(2_p)2. hl other words, we 
have çp tg Ap_2+c(2_p)2,1 p bv. Proposition 2.2. N-te that thc hvpohesis p _> 1 is 
used here to ass,re that p - 2 + c(2 - p) > - 1. Now, «hoose s E [0, 1) su«h that 
sp - 2 + c(2 - p)Z > -1 and put « = sp - 2 + c(2 - p)2. Th('u, wc bave s < +2 
Also. since 4 p 
• p-2+c(2-p)=,l "-4Pa,s bv. (1.1), we havc çp  4 v,s. [] 

The next examplc shows that the lower bomd s > +-- in Thcorem 3.3 is sharp 
when o > -1. 

Exanlple 6.3. Let p > 1, ,. > -1 and put s -- +2 Then. there exists a 
P 
holomorphi« self-nmp ç of D with ç(1) = 1 su«h that C' • APo, s -- APa,, is b(mnd('d 
but ç does hot have angula.r derivative a.t z = 1. 

Pro@ Let ç = çn be any lens lnap. Note that a + (1 - s)p > -1. Thus, as in the 
proof of Proposition 5.4, we have 
m that @ " A,  A,, is bo,mded bv Theorem 2.60). Clearly, ç does hot have 
an angular dcrivative a.t 1. 

The next example shows that the upper bound for O(P) in Theorem 5.5 is sharp 
when Cil > -- 1. 

Example 6.4. 
O I > --1 and 

Let pi, si, aj be as in the hypotheses of Theorem 5.5. Assume 

Then, f o çu  4 2 

PI (02 + 2 -- 
(6.4) p2(o + 2 - SlPl) < ri < 1. 
for some f G 4 P 
 Cq ,S I " 

Pro@ Let ç = ç,. Choose 0 < a < 1 such that ]ç(a)l _> 1/2. Also. by using (6.4), 
choose « > 0 sufficiently small so that 

(6.5) (2 + o2)/p 2 -- .» + a 
- <]<1 
(2 + O1)/p -- S 

Now, consider the test fimction f(z) = log(1 -ç(a)z). Let k _> s be a positive 
integer. Then we have 4 pI  A pi 
« C,,I..S1 tl_]_(__S1)p,] by (1.1). Therefore, bv Proposition 



2854 B. CHOE, H. KOO, AND W. SMITH 

2.2, (2.3), (6.2) and (6.5), 
(6.6) < 
On the other hand, for e2 > -1, 

and thus by (6.2), 
(6.7) 

(1 - I(a)l 2 ) 

[BB] 
[CM] 
[D] 
[HKZ] 

[Ma] 
[PI 

[Pd] 

R EFERENCES 

F. Beatrous and J. Burbea, Holomorphic Sobolev spaces on the ball. Dissertationes Math- 
ematicae, CCLXXVI (1989). 1-57. MR 90k:32010 
C. Çowen and B. IklacCluer, Composition operators on spaces of analytic functions, CRC 
Press. Boca Raton. FL, 1995. MR 97i:47056 
P. Duren. Theory of H p spaces, Pure and Appl. Math., Vol. 38. Academic Press, New 
York. 1970. MR 42:3552 
J. Garnett. Bounded analytic functions, Pure and Appl. Math., vol. 96, Academic Press, 
New York, 1981. lkIR 83g:30037 
P. Jones and N. lklakarov, Density properties of harmonic measure, Armais of Mathemath- 
ics, 142 (1995), 427-455. lkIR 96k:3002ï 
H. Hedenmalm, B. Korenblum and N. Zhu. Theory of Bergman spaces, Graduate Texts in 
Math., vol. 199, Springer. New York, 2000. MR 2001c:46043 
D. Luecking, Forward and Reverse Carleson inequalities for functions in Bergman spaces 
and their derivatives, Amer. J. Math., 107 (1985), 85-111. MR 86g:30002 
D. Luecking, Multipliers of Bergman spaces into Lebesgue spaces, Proc. Edinburgh Math. 
Soc., 29 (1986), 125-131. MR 87e:46034 
B. MacCuer and J. H. Shaprio, Angular derivatives and compact composition operators 
on the Hardy and Bergman spaces, Canad. J. Math., XXXVIII(4) (1986), MR 87h:47048 
878-906. 
K. Madigan, Composition operators on analytic Lipschifz spaces. Proc. Amer. Math. Soc., 
119 (1993), 465-473. MR 93k:47043 
J. Miao, A property of analytic functions with Hadamard gaps, Bull. Austral. Math. Soc., 
45 (1992), 105-112. MR 93c:30060 
Ch. Pommerenke, Boundary behavior of conformal maps, Grundlehren der Mathema- 
tishen Wissenschaften, vol. 299, Springer-Verlag, Berlin. Heidelberg, New York, 1992. 
95b:30008 
W. Rudin, Funct,on theory in the unit ball of C n, Grundlehren der Mathematishen V'is- 
senschaften, vol. 241, Springer-Verlag, New York, 1980. MR 82i:32002 



COMPOSITION OPEPATORS ON SOBOLEV SPACES 2855 

[R2] 
[Sh] 
[Sml] 
[Sm2l 
[SYI 
[ST] 

W. Rudin. Real and complex analysis, McGraw-Hill, New York, 1987. M1R 88k:00002 
J.H. Shapiro, Compact composition operators on spaces of boundary-regular holomorphic 
functions, Proc. Amer. Math. Soc., 100(1) (1987), 49-57. Ml 88c:47059 
W. Smith. Composition operators between Bergman and Hardy spaces, Trans. Amer. Math. 
Soc., 348(6) (1996), 2331-2348. MR 96i:47056 
W. Smith, Compactness of composition operators on BMOA, Proc. Amer. Math. Soc., 
127(9) (1999}, 2715-2725. MIR 99m:47040 
W. Smith and L. Yang, Composition operators that improve integrability on weighted 
Bergman spaces, Proc. Amer. hlath. Soc., 126(2) (1998), 411-420. hIIR 98d:47070 
.1. Shapiro and P. Taylor. Compact, nuclear, and Hilbert-Schmidt composition operators 
on H 2, Indiana Univ. Math. J., 23(6) (1973), 471-496. MR 48:4816 

DEPARTMENT OF h[ATHEMATICS, [OREA UNIVERSITY, SEOUL 136--701, [OREA 
E-mail address: choebr@math, korea, ac. kr 

DEPARTMENT OF I[ATHEMATICS, KOREA UNIVERSITY. SEOUL 136-701. KOREA 
E-mail address: koohw@math, korea, ac. kr 

DEPARTMENT OF ]IATHEMATICS, UNIVERSIT , OF HVAII. HONOLULU. HAV:«II 96822 
E-mail address : wayne@math.hawaii.edu 



TRANSACTIONS OF TtiE 
AMERICAN MATHEMOETICAL SOCIETY 
"volume 355, Number 7, Pages 2857-2883 
S 0002-9947(03)03248-3 
Article electronically published on Match I9, 2003 

DISTRIBUTIONS OF CORANK 1 
AND THEIR CHARACTERISTIC VECTOR FIELDS 

B. JAKUDCZYK AND M. ZHITOMIRSKII 

ABSTRACT. oEre prove that any l-parameter family of corank 1 distributions (or 
Pfaff equations) on a compact manifold 5I ' is trivializable, i.e., transfornlable 
to a constant family by a fanily of diffeonmrphisms, if ail distributions of the 
family have the saine characteristic line field. The characteristic line field is a 
field of tangent lines which is invariantly assigned to a corank one distribution, 
If is defined on lI n, if , = 2/,', or on a subset of AI" called the Mmtinet 
tlypersurface, if n : 2k -I- 1. Our second main result states tllat if two corank 
one distributions have the saille characteristic line field and are close to each 
other, then they are equivalent via a diffeOlnorphisln. This holds tmder a weak 
assumption on thc singularities of the distributions. The second result ilnplies 
that the abnorlnal curves of a distributiol determine the equivalence class of 
the distribution, among distributions close to a given one. 

O. INTRODUCTION 
The well-known Gray theoreln [G] states that any 1-parameter fmnilv of contact 
structures on a compact manifold ,l).a+l is trivializable, i.e., transformable to a 
constant falnily by a fmnily of diffeomorl)hislnS. Our first main result generalizes 
this theorem to he case of 'singular contact structures", for which the contact 
condition is satisfied on a dense subset of M, and to cora|lk one distributions 
on llmnifolds of even dilnension. In these cases the famil.v of distributions has to 
preserve, when the parameter changes, a characteristic line fiel(l. The characteristic 
lie field is a field of tangem lines which is invariantly assigned to a corank one 
distribution (it is defined on M  if  = 2/,', or on a hypersm'face of/1I  if  = 2k+l). 
Our second main result states that if îwo corank Olle distritmtions have the Saille characteristic line field raid are close to each other, then they are equivalent via a 
diffeomorphism. It lneans, in particular, that the chm'acteristic line field contains 
COlnplete information about the geometry of singularities of the distribution. Our 
results hold under a weak &ssumption on the distributions, called conditiol 
saying that the depth of a characteristic ideal of the distribution is nondegenerate 
at singular points of the characteristic line field. 

Received by the editors January 9, 2002 and, in revised form, September 4, 2002. 
2000 Mathematics Subject Classification. Primary 58A17; Secondarv 53B99. 
Key wovds and phrases. Pfaff equation, equivalence, contact structure, quasi-contact struct ure, 
singularity, invariants, line field, homotopy method. 
The first author was supported by the Committee for Scientific Research (KBN), Poland, grant 
2P03A 03516. 
The second author was supported by the Fund for the Promotion of Research at the Technion. 

(2003 American 1Mathemalical Socict3 



2858 B. JAKUBCZYK AND M. ZHITOMIRSKII 

Let us state out second result in the simple case where the characteristic line field 
does not have singularities. Assume  = 2k >_ 4 and consider a smooth differential 
1-forln ce on /1I n. Let (w A (dw)k-1)(p)  0 for ail p • /1I n. Such a forln w defines 
the distribution A= kerce, called the quasi-contact structure defined by w. Then 
dw(p), ïestricted to the distribution Ap) = kerw(p), is of nmximal possible tank 
2k - 2 aud has 1-dimensional keïnel. We define the characteristic line at p by 
L v = ker dw(p) 
The characteristic line fleld L = L(w) is the fiekl of characteristic lines p -=, L v 
on /11. The folloxving fact is a special case of our Theoreln 1.2, the case where 
singularities are absent. (It is also a special case of a theorem in [MZh], Appendix 
A, concerning corank 1 distributions of constallt class.) 
Theorem 0.1. Let A be tbe quasi-contact structure on a compact orietable man- 
ifold III 2 defined by a 1-form w. and let & be a 1-form such that L(&) = L(w). If 
& is sujïjïciently C2-close to ce, then there exists a diffeomorphism of M 2 sending 
ï = ker& fo A. 
If a compact orientable manifold ]I 2a adnfits a quasi-contact structure defined 
bv a global 1-form w, then its characteristic line field is generated by a nonvanish- 
ing global vector field and the Euler chmacteristic of/I 2a is eqnal to zero. Thus 
a manifold /I 2a with uonzero Euler dmracteristic adnfits only corank one distri- 
butious with singular characteristic line field. Eveu if/I 2a admits a quasi-contact 
structure, siugularities nmv appear naturally when restricting a corank one distri- 
bution to a subnmnifold of even dimensiou. Therefore, it is natural to ask whether 
Theorem 0.1 holds in the preseuce of singularities. Theorem 1.2 in Section 1 gives 
a positive answer under assumption (A), saying that the singularities of the char- 
acteristic vector field bave a natural dcpth (and codimensiou, in the analvtic case). 
This assumption excludes singularities of iufinite codimension. 
To state a siufilar result in the case n : 2k + 1. we introduce the set of points 
where w does hot satisfy the contact conditiou: 
S = {p • IIl 2+ • (w A (dw)k)(p = O}. 
This set is called the Martinet hypersurface. The Martinet hypersurface is the set 
of zeros of the flmctiou 
H = ce A 
where {2 is a volulne forln. If S is elnpty, i.e., ce is a contact 1-fonn on 3I 2+, 
then A = kerw is globally equivalent to any distribution , sufficiently close to 
A. This follows frolu the theoïem of Gray lnentioned above. Assume now that 
S is louempty. \Ve shall call A = kerce a Martinet distribution if it satisfies the 
following two conditions: 
(a) dH(p) ¢ 0 for all p • S (then S is smooth), and 
(b) As = kerces is a quasi-contact structure on S, where ws is the pullback of 
wtoS. 
At each point of S we can define the characteristic line 
Lp = kerdws(P)]ks(p), p • S. 
The characteristic line field L = L(w) for a Martinet distribution & is the field of 
tangent lines p -- Lp on S. It has no singularities. The following fact is a special 
case of out Theorem 1.4. 



DISTI:IIBUTIONS OF COI:IANK 1 2859 

Theorem 0.2. Let A = kerw be a Martinet distribution on a compact orientable 
manifold 1112k+1 . If the Martinet hypersurfaces ad the characteristic line fields of 
A and/ = kera5 are the saine and & is sujïficietly Ca-close to w, t]en there exists 
a diffeornorphisrn of tle rnanifold sending A to A. 
Martinet distributions fonn a restrictive class of corank one distributions (L may 
have singularities). In particulm, any Martinet distribution on M 3 has the Mmtinet 
hypersurface S, which consists of two-dimcnsional tori (each connected component 
of S has zero Euler characteristic, since S admits a l-dimensional foliation defined 
by L). Out Theorem 1.4 will generalize Theorem 0.2 to thc case of glmral corank 
one distributions and PfaIf equations on/ll 2k+1. 
The appemauce of the characteristic line field L as one of the main invariants 
of corank one distributions A has the following history. The first to study L -cas 
J. Martinet in [Mari for the simplest occurring singularities of X on IR 2+ (when 
L has no siugularities). J. Martiuet also started to study typical singularities of 
L in the 3-dimensional case. These singularities were roughly classified in [JP], 
where the existence of a modulus in the classification of characteristic line fields 
was shown. It was proved in [Zhl] by obtaining a normal form for A that this 
modulus is the only iuvariaut of A. This gave thc complete local classification of 
germs of generic 2-distributious ou 3-manifolds, with the characteristic line field as 
the complete invariant. 
In the book [Zh2] the second author gave a classification of finitely determined 
singularities of corauk one distributious and PfaIf equations on mauifolds of any 
dimension. In this case again the chm'acteristic line field L is a complete invariaut. 
This justifies, to a large extent, lnaking the followiug 
Conjecture. In the space of germs at 0  IR ' of corank one distribations on IR ', 
there is an ezceptional set of infinite codirnension such that for any two distributions 
Ao and As away front this set, with the saine characteristic line field L, there ezists 
a local diffeomorphism op : (IR ', O) -- (IR ', O) redacing X to A0. 
In a weaker form, with n = 3, this conjecture has already appeared iii [JP] and in 
a letter from J. Mmtinet to the second author (in 1989). In this case the conjecture 
tan be deduced from the results in [JZh2], [JZhl]. The exceptional set consists of 
germs that do hot satisfy the assumption (A) or that do not have the property of 
zeros (sec Section 1). In [JZh2] we proved that away rioto the exceptional set the 
restriction of the distribution to the Martinet hypersurface is a complete invariant 
for any n = 2k + 1. If k = 1, then the restriction eau be identified with the 
characteristic line field. 
The results of the present paper concern global corank one distributions close to 
a fixed distribution. They also deal with families of distributions. Iii this setting we 
eliminate some of the diftàculties iii the above conjecture which are due to non-close 
germs and the necessity of preserving a fixed point (the source of the germ). 
Our transition to the global approach was inspired by the results in [Gol] and 
[MZh, Appendix AI. In [Gol] it is proved that a fmnily of Engel structures Et on 
a 4-manifold is trivializable provided that the characteristic line field of Et does 
not depend ou t. The result in Appendix A of [MZh] states that two close global 
corank one distributions of any constant class (iu the Cartau-Frobenius sense) are 
diffeomorphic provided that they have diffeomorphic characteristic foliations. This 
is a generalization of the Gray theorem. All these results apply to objects without 



2860 B. JAKUBCZYK AND M ZHITOMIRSKII 

singularities, whereas in the present paper we allow any singularities except certain 
ones of infinite codimension, excluded by the assumption (A). The presence of 
singularities leads to the main ditïficulties iii ollr proofs. In Section 2 we explain 
tllat, the assumption (A) is natural and give examples showing that it caimot be 
weakened. 
In [JZhl] and [JZh2] we ohtained local realization theorems {for gerlns at a fixed 
point }, t heorems characterizing the set of all possible characteristic line fields if r = 
3. Combined with the reduction theorems, thev lead to a number of applications 
including classification results. In this paper we leave aside the diflïcult task of 
obtailfing global realization theorems. The absence of such theorems restricts, at 
present, the possibility of drawing imlnediate iuteresting conclusions concerning 
global classification of corank one distrilmtions. 
Note that the assmnption of closeness which appears in Theorems 0.1 and 0.2 
{and later in Theorelns 1.2 and 1.4} is essential for out results. Already two contact 
structures that are far ff'oto each other are, in general, hot equivalent. A classifica- 
tion of contact structures is known only on certain 3-diluensional manifolds, sec e.g. 
[TEG]. We hope that our results can be used in glolml contact or quasi-contact ge- 
»metrv for studying singularities of 1-forms which appear when two global contact 
(or quasi-contact) structures are joilmd by a path. 
There are natural consequences of our results concerning characteristic curves of 
a distrilmtion, also called sigular curves or abormal cu.rves in sub-Rielnannian 
geometly, all(l t]le geolnetry of distril,utions (cf. lAI, [BH], [LS], [Mon]). These 
curves coincide in out case with t]le integral curves of the characteristic line field. 
Under thc assumptions of Theorerns 0.1 ad 0.2, as well as those of Theorems 
1.2 and 1. it Se«tiot 1. the singular curves of a «otanC" 1 distribution &termine 
the equivalece class of the distribution, amog ditributions close fo a gi-ven one 
it the C œ topolog. 

1. TATEMENT OF tlESULTS 

We will deal with Pfaff equations, v«hich are more general objects than (coori- 
ented) corank one distributions. Let. M u &,note a compact, orientable, Hausdorff 
lnanifold of dimension n _> 3. Bv definition, a Pfaff equation is a set of differential 
l-ri»tins on M' generated, as a module over the ring of fimctions, bv a single 1-form 
w. In other v«ords, a Pfaff equation is a 1-form on .1I ' defined up to lnultiplica- 
tion bv a nonvanishing flmction. We dcnote the Pfaff equation by P = (w). If w 
vanishes at no points of M', then (w) can be identified with the field of kernels of 
w--a coorientable hyperI)lalm fie]d in TIIl '. In general, a Pfaff equation is a more 
general object, siuce we do not exclude the possibility of w vanishing at some points 
of the manifold. 
Ail objects in this paper will belong to a fixed category which is either C  or 
real analytic C ". 
The case  = 2/,'. To any Pfaff equation P. and in particular to any cooriented 
corank Olle distribution, one can associate the characteristic line field. 

Definition 1.1. If n is even. n = 2/,', then anv vector field X defined by the relation 

XJ = to A (dw) k-l, 



DISTF{IBUTIONS OF COIRANK 1 2861 

where t is a.lV vohune form, a.ll(l ce any geuerator of P, is called a cbaracteristic 
vector field of P. The line field on In generated by X, i.e., the lnapping 
is called the «haracteristic l{e fie[d. Thc characteristic line field will be denoted 
by L or L(w) or L(P}. The set {p  I  • X(p) = 0} is callcd thc set of singular 
points of L and dcnoted by Sing(L). 
It is easv to check that anv two characteristic vector fields differ bv multipli- 
cation by a nonvanishing fuucIi(m, and consequently thc characteristic line field is 
invariant.ly related to the Pfaff equation P, i.e.. the choice of the generator  of P 
and the volmne form [ is irrelevant. Note that if ( A ()k-1)(p)  0. then the 
definitions of Lp and L coinçide with those given in the introduction. 
In the presence of singularities we need ail invariant that describes a "degree of 
degencration'" of singular points of a characteristic vector field X (for n odd il was 
introduced in [JZh2]). First we define i/il invariant that is slightly stronger than 
the set Sing(L). 
Let p  Sittg(L ). Thc loc«d chwracteristic ideal Ip at a point p G .I 2k of a Pfaff 
equation P is the idcal Iv(X ) in the ring of functioli gerlns at p. gcnerated Iv the 
coefficients (i 1 ..... a,, of a characteristi« vector field X of P. in some cooMinate 
svstem near p. It is easv to sec that the ideal Ip is invarialitlv related to the gerln 
at p of P = () (the choices of a chara«teristic vector field and of a local coordinate 
system are irrelevant). The germ at p of the set Si,(L) is the zero set of 
Definition 1.2. If n = 2k and p  Sittg(L), then wc define dp(P) = dp(X) as 
«(P) = d,,pth IdX). 
Recall that the depth of a proper ideal I C R of a ring R is the maximal length 
of a regular sequence of elements in I. A sequence ai ..... af  I is called regular if 
al is hot a zero divisor in and. for any i = 2,..., r. the element ai is hot a zero 
divisor in the quotient ring R/(a,.... ai-l), where (al ..... (ri-l) denotes the ideal 
generated bv ai,..., (i-1. Bv definition, depth R = . 
Remark. In the alialvtic category, dp(P) is equal to the codimelIsion in Ç' of the 
germ at p of the set of complex zeros of the ideal Ip (i.e., the zero level set of 
the ideal generated by the complexification of the generators of Ip in some local 
coordilmtes). This follows ff'oto the fact that the complexification does hot change 
the depth of an idem of analytic fimctioli germs (cf. e.g. [E]) and ffoto the equality 
of the depth(I) and the codimension of the analvti« set of zeros of I for ara" ideal 
I of holomorphic function gel'lUS. 
XVe introduce the followilig crucial condition: 
(A) dp(P)  3 for anv point p  Sing(L ). 
This condition is rather weak, in particular generic, as will be explained in Section 
2. 
The following theorems hold in the cat.egories C  and C , with AI a compact 
orientable manifold. 
Theorem 1.1. Let Pt. l  [0, 1], be a famil of Pfaff equations ot ;I . k  2. 
that satisfies the followin9 coditions. 
(a) All P, define the saine characteristic line field L = L(Pt). 



2862 13. JAKUBCZYK AND bi. ZHITOMIRSKII 

(b) Ail Pt satisfy condition (A). 
Then there exists a family (Pt of diffeomorphisms of ]il 2k sending Pt to Po. 
Theorem 1.2. Let Po = (Wo) and P1 = (wl) be Pfaff equations on ]il 2¢. k >_ 2, that 
have the same characteristic line field L = L(Po) = L(P). Assume that condition 
(A) holds for Po. Then there exists a diffeomorphism (P sending P to Po provided 
that w is suJficiently close to Wo in the C œ topology. 
In the above theorems as well as in Theorems 1.3 and 1.4 below, all objects are 
in the saine category C œ or C °, including regularity with respect to the paranmter 
t. The diffeomorphism in Theorem 1.2 can be taken Coe-close to the identity. 
Remark (Closeness of w to w0)). In nmny cases one can present a number r < OE 
(depending on Po) such that the closeness in the C  topology in Theorem 1.2 tan 
be replaced by closeness in the Cr topology. See Theorem 0.1 and Theorem B.2 in 
Appendix B. 
The case n = 2k + 1. The most basic invariant of a Pfaff equation P = (w) on 
/I 2+ is the set 
S = {p e //12k+ : (w A (d)k)(p) = 0}, 
called the Martinet lypersurface, which consists of points at which w is not a contact 
form. This set, invariantly related to P, is the zero level of the function 
/4 =  A (d)/f, 
where ç is a volulne form. 
The ideal (H) of the ring of functions on ]il 2k+, generated by H, is also invari- 
antly related to P. It is called the Martinet ideal. 
The characteristic line field of P = (w) on/12k+ is defined on the set S. 
Definition 1.3. Anv vector field X on/i2+ satisfying the relation 
X]i = o3 A (dw) k-1 A dH mod (H). 
where w is an 3, generator of P, and H is an3" generator of the Martinet ideal, will 
be called a characteristic vector field of P. The line field on S = {H = 0} defined 
by the relation 
p-- L» = {eX(p), a e OE}, p e S. 
is called the characteristic line field of P. It will be denoted by L or L(w) or L(P). 
The set of singular points of L is defined as Sing(L) = {p e M 2+1 • H(p) = 
o, x(p) = 0}. 
Above and in the test of the paper the equality of two objects (vector fields, 
differential forms) mod (H) means that their difference is divisible over H in the 
space of objects of the saine category. To check that the line field defined above is 
tangent to S, note that the definition of X implies that XJdH = 0 at any point 
of S = {H = 0} and that X vanishes at any point of S at which the l-form dH 
vanishes. Thus X(p) Ç TpS at any point p  S at which S is a slnooth hypersurface 
and X vanishes at all other points. It is easy to check that the characteristic line field 
is invariantly related to P, i.e., the choices of the generator w of P. the generator 
H of the Martinet ideM, and the volume form Q are irrelevant. 
Note that in the case of the Martinet singularity (p  S and (, A (d,) k- A 
dH)(p) 7(= 0) the definitions of Lp and L coincide with those given in the introduc- 
tion. 



DISTIIBUTIONS OF CORANK 1 2863 

In order to deal with deeper singularities of P, namely those allowing singular 
points of L, we introduce out invariant dp(P) in the case of odd n as follows. 
Let p E Sing(L). The local characteristic ideal Ip at a point p G /_/2k+l of a 
Pfaff equation P is the ideal Iu(H, X) in the ring of function germs at p generated 
bv the germ Hp of a generator of the Martinet ideal and the coeflïcients 
of a characteristic vector field X of P, in some coordilmte system near p. The ideal 
I u is invariantly related to the germ at p of P = (w) (the choices of a characteristic 
vector field, a generator of the lklm'tinet ideal, and of a local coordinate system are 
irrelevant). The germ at p of tlm set Sig(L) is the zero set of Ip. 
Definition 1.4. Il:  = 2k + 1 and p G S«ug(L), then we define 
du(P) =du(H,-\" ) 
as the maximal length of a regular sequence in the characteristic ideal Ip(H. 
starting with the germ H u as the first element. 
Remarks. (a) In Noetherian rings all nmximal regular sequences in I are of the 
saine finite length, lk[oreover, any regular sequence can be completed to a maximal 
regular sequence. This implies that in the analytic category, independently of the 
parity of n, we have 
du(P) = depth(1 u). 
(b) Similarlv to the case  = 2k, in the analytic category, du(P ) is equal to the codi- 
mension in C of the germ at p of the set of complex zeros of the complexification 
of the idem 
To formulate our reduction theorem for the most general case, we need two prop- 
erties of the Martinet ideal (H): the property of zeros and the extension property. 
Definition 1.5. The Martinet ideal (H) has the property of zeros if for any p 
S = {H :- 0} the ideal in the ring of ail function germs at p generated bv the germ 
Hp of H at p coincides with the ideal in the saine ring consisting of function germs 
vanishing on the germ at p of the set S --- {H -= 0}. 
The property of zeros allows us to identify the hlartinet hypersurface S 
0} with the hla.rtinet ideal. In the case of germs this follows from the definition. 
Examples where the property of zeros is violated at a point p include: 
(a) H u = HîH2. where Hx, H2 are function germs and H(p) = 
(b) H u is equivlent to r 2 = xî +-.-+ x; 
(c) Hp is a fiat germ (i.e., the Taylor series of H at p is zero): 
(d) H u is a zero divisor in the ring of all germs at p. 
In case (c) the property of zeros is violated, since u = r-2Hu is smooth and 
has the same germ of zeros as H u but p ¢_ (Hp). Note that (d) is a particular case 
of (c). 
The local version of the property of zeros (Definition 1.5) implies the global 
version: if a function f on/I vanishes on the set S = {H --- 0, then f belongs to 
the ideal (H). This follows from the fact that division by H or by the germ H u is 
1 
unique (by (d) the germ H u is hot a zero divisor). 

In the Coe-category the global and local versions of the property of zeros are equivalent (this 
follows from the partition of unity). In the real analytic category they are equivalent provided 
that the sheaf of functions vanishing on S = {H = 0} s coherent. In the proof "global implies 



2864 B. JAIxUBCZYK AND M. ZHITOMIRSKII 

We also need the e.rtension propert. of the Martinet hypersurface S = {H = 0}. 
Denote by Çoe(l) the Fréchet space of smooth fimctions on AI, equipped with 
the topology of convergence together wfih all derivaiives. Let Ç(,l. S) denoie 
its closed subsl)ace of functions that vanish on S.  define the space of smooth 
funclions on S as the quotienl Déchet space C(S) = COE(kI)/Ç(:II. S). 
Definition 1.6.  sa,y that S has the ea'tensio propert if there exists a COlltillU- 
o,,s linear operator A " Ç(S)  ÇX(kl) such that A(f)[s : f for ail f G COe(S). 
The extension property automat.ically holds in the Ç category, since it holds 
for anv analytic subset S of Al (see [BS] tbr a more general extension theorem). It 
also holds if wo assmne that (H) h, locally aromid any point p G S. a generator 
that is analytic in some coordinate system. 
The fifllowing theorems hold in the categories Coe and Ç, with M compact and 
oriental)le. 
Theorem 1.3. Let Pt.   [0.1], be a famil# of Pfa equatious on M +. k  1. 
that satzsfles the follouring conditions. 
(a) All Pt bave the .saine Mn«'tinet hyper.surface S, which bas the etesion prop- 
erty, ad their Martitct ideals have the propertl of zeros (and cousequetly are the 
(b) All P define the saine chamcteristie line field L = L(Pt). 
(e) Ail Pt satisf9 condition (A). 
The there eists a .family ( qf diffeomorphimns of ./a+ sendin 9 P fo Po. 
Remark. Recall that the extension prol)ery of S holds autonmticallv in the C  
caegory. % conjecture that in the C category the extension property in Theorem 
1.3 also can be omitted. Our proofs show that this is so if le familv Pt h a 
generator  that is polynonfial in t. 
Theorem 1.4. Let Po = (o) ad PI : () be Pfaff equations on M +, h  1. 
u,hieh bave the saine Martinet hypersurface S = S(Po) = ,ç(P) and the saine 
characteristic lie fleld L : L(Po) = L(P). Assune that the Mainet ideal of 
Po bas the property of zeros and Po satisfies condition (A). Then there eists a 
diffeomorphism • sending P fo Po, provided that ' is su«iently close to 'o in 
the C  topology. 
Remarh (Closeness of  to 0)- As in the even-dimensional ce, often one can 
present a mmfl)er r < . (depending on P0) such that the closeness in the C  
topology in Theoreln 1.4 ca. t)e replaced by closeness in the C topology. Sec 
Theorem 0.2 and Theorem B.2 in Appendix B. 
The contents of the rucher .'ections. In Section 2 we explain whv the condition 
(A) is natural and give examples showing that it. cammt be veakened. The conse- 
quences of condit.ion (A) are explained in Section 3 and Appendix A: the condition 
(A) implies certain global division properties of a characteristic vector field. Sec- 
tion 4 contains auxiliarv algebraic statements, which also will be used throughout 
the proofs. Using the division properties and these algebraic staements, ve prove 
Theorems 1.1 and 1.3 in Sect.ions 5 and 7, respectively. The proofs of these the- 
orems are based on the honmtopy method, according to which it suffices to prove 

local" one should use Cartan's Theorem A in [C], which says that any local section of a coherent 
analytic module belongs to the module generated bv global sections. 



DISTRIBUTIONS OF CORANK 1 2865 

the solvabilitv of the equation 
(HE) Lz, wt q- htcot q- d7 = 0 
with respect t.o a family of vector tiens Zt and a family of fimctions ht (here Lzw 
denotes the Lie derivative of w along Z). Tllen the familv t of diffeomorphisms 
obtained by integrating the familv of vector fields Zt, 
dt 
- Zt(Ot), o = id, 
dt 
transfonns the Pfaff equations (wt) into (w0): 
êt w = tWo, 
where 't = exp(- ]ô t.ds) and t = ht o et- In what follows thc equation (HE) 
will be called the homotopg equatio or bomological equation. 
Theorenls 1.2 and 1.4 are proved in Sections 6 and 8 I)y reduçtion to Theorems 
1.1 and 1.3. In these sections we show that if Pfaff equations Po and P satisfv the 
assumptions of Theorem 1.2 or 1.4. then tlwre exist generators w0 of P0 and  
of P1 suçh that the path of Pfaff equations Pi generated bv wt = w0 + t(w - w0) 
satisfies Ihe assmnptions of Theorcm 1.1 or Theorem 1.3. 
In Al)Iwndix B we present certain topological properties of linear operators re- 
lat.ed to the 5Im't.inet ideal and /he characteristic ideal. W also show a vav of 
transition from the assmnption of C-closeness of w to 0 in Theorems 1.2 and 
1.4 to the C-closeness with a certain r < ,. In the simplest cases this wav leads 
to Theorems 0.1 and 0.2 in the lntroduçtion. 

_9. NECESSITY OF CONDITION (A) 
In this section we explain whv the condition 
(A) dv(P ) >_ 3 
is natural, and we give examples showing that this condition cmmot be weakened: 
if depth dv(P ) = 2. then out theorems are hot truc anvmore. 
Fix a point p E AI a and denote by J the space of/-jets of 1-forlns at p. The 
condition that p is a singular point of the characteristic line field L, i.e. p G Sing(L), 
is the condition 
(u3 /k ((]oE)k--1)(p) = O, 

if n = 2k, and 

(,d A (dw)k)(p) = 0. (,/ (dc6) k-1 A dH)(V) -=- O. 
ifn = 2k+ 1. It involves the /-jet at p of a generator, of P, where i = 1 if 
z is even and i = 2 if n is odd. This condition distinguishes a certain subset 
of J--tlle space of/-jets of w at p. It is hot difficult to see that for any parity 
of n this subset is a stratified submanifold of codimension 3 (see [Mari, [Zh2], or 
[JZh2] for lnore details). Consequently, for generic , the set Siz9(L ) is either 
empty or a submmlifold of AI r of codimension 3. In the real analytic category (and 
conjecturally, in the smooth category too) the set of 1-forlns , violating (A) bas 
infinite codimension in the space of all 1-forms on AIr: see [JZh2], Proposition 3.4 
and Theorem A2 (Appendix 2). 
The following examples show that the conditioll (A) cannot be replaced by the 
condition dp(P) OE 2. In these examples dira M = 4 and diln M = 5. They can be 



2866 B. JAKUBCZYK AND M. ZHITOMIRSKII 

extended to higher dimensions. We have not round an example in the 3-dimensional 
case, but we believe that such an example exists. 
Example 1. Consider the family of Pfaff equations on the 4-torus T 4 generated 
by 1-forlns 
#t = dO + (sin 03 sin 04 + t)d02. 
The characteristic vector field Xt is the sanie for ail t: 
0 0 
Xt = X0 = cos 04 sin 0 3 33 COS 0 3 sin 04 004" 
The set S of singular points of X0 is the union of 8 disjoint 2-dimensional tori (4 
of thcm are described by the equations 0a, 0a E {7r/2, -7r/2}, and the other 4 by the 
equations 0a, 0a E {0,-r}). The codimension of S is 2; therefore depth Ip = 2. The 
restriction of (t) to any of these 2-dilnensiolml tori is a Pfaff equation generated 
by a 1-form ct = dol + ( + t)d02, where 5  {0, 4-1} depending on the torus. 
This Pfaff equation eau be identified with the vector field Vt : (5 + t)O/O01 - 0/002 
defined up to nmltiplication by a nonvanishing function. It follows that the phase 
portrait of I/ on the tortis is invariantly related to (at). It is well known that 
the equivalence of the phase portraits of 1  and t with a fixed 5 implies ri = t2 
provided that t2 is close to tl; see [ArI1] (the parameter t corresponds to the rotation 
number). Therefore the parameter t of the family Pt is a modulus (a parameter 
varying continuously and distinguishing nonequivalent Pfaff equations). 
Example 2. Consider the family of Pfaff equations on the 5-torus 
T(O, 02, ¢1, ¢2, 
generated by 1-forms 
t = (A(0, 02) + Bt(01.02, ¢2)) d¢ + C(01.02)d¢2 + des, 

where 

A(01,02) = 3(sin 01 + sin 02), 
Bi(01,02) = t sin¢2 (1 - cos(0 - 02)), 
C(O. 02) = cos 01 + cos 02. 
A simple calculation gives that t A (dt) 2 = sin(01 - 02)-Qt" fL where f is a volume 
form and Qt is a family of nonvanishing functions on T 5, if t  [-1.1]. Therefore 
the Martinet ideal is the smne for ail t; it is generated by the function sin(01 - 02). 
The Martinet hypersurface consists of two disjoint 4-tori: 
s= îu', î = {o2 =ol}, ' = {o2 =ol + }. 
Since the function Bi vanishes on the torus T14, the restriction of (ut) to T14 does 
hot depend on t. The restriction of (aJ) to the torus T_(¢1, ¢2, ¢3.0) depends on 
t: it is the Pfaff equation (ct), where (ct) = 2t sin ¢2d¢ + d¢3. The characteristic 
vector field of (t) restricted to T is the characteristic vector field of (ct). It is 
2t cos ¢20/001. Assume that t #- 0. Then the characteristic line field does not 
depend on t. The set of its singular points is the union of two disjoint 3-tori T:ï:, 
given by the equations ¢2 = 4-7r/2. The restriction of (at) to T, 3 (or. the saine, the 
restriction of (ct) to T:ï:) is the Pfaff equation of the form (/3t),/3t = 2t5dçbl + d¢3, 
  {-1, 1}. Consider the vector field Vt : -0/0¢1 + 2t0/0¢3 on the 2-torus 
T 2 = T2(¢,¢3). It is easy to sec that the Pfaff equations (,3) and (,3t2) on the 



DISTRIBUTIONS OF CORANK 1 2867 

3-torus are equivaleut if and only if the phase portraits of t 1 and r 2 on the 2- 
torus are equivalent. As in the previous example, this is so if and only if ri = t2 
provided that tu is close to tl. Therefore the paralneter t of the family (wt) of Pfaff 
equations is a modulus, although these Pfaff equations have the saine Martinet ideal 
(satis[ving the property of zeros) and the saine characteristic line field. The reason 
for that is the violation of the assulnption (A)- the dcpth of the characteristic idem 
is equal to 2 instead of 3. 

3. CONDITION (A) AND DIVISION PROPERTIES 
In this section ve explain implications of condition (A) which will be essential in 
fm'ther proofs. The main implicatious are the following global division properties 
of a characteristic vector field X. As belote, we work in the C œ and C  categories. 
Proposition 3.1.a. If a Pfaff equation P = (w) satisfies condition (A). then any 
characteristic vector field X of P bas the following division properties. 
(i) If n is even. then for any vector field " and any r-form u on ]I  with 
r = n-1 or r =  - 2 the equality 
X]u = 0 implies u = 
for an (r + 1)-form p on M', and the eqmlity 
X A t = 0 implies I = fX 
for a function f on M n. 
(ii) If n is odd. then for any vector field }" on 1I ' and any ( - 1)-form u on 
M' the equality 
xJu=o mod(H) imvlies u=xJg mod (H), 
for an n-form # on M , and the equality 
XAI=0 lnod (H) implies Y=fXmod (H) 
for a function f on M'. Here (H) is the Martinet ideal of P, and we assume that 
( H) bas the property of zeros. 
This proposition is a corollary of a general theorem in [DJ] on division properties 
of the interior product with a section X of a vector bundle (see Appendix A for the 
proof). 
We also need a division property with parameters. In the next and all further 
st.atements in this section a 1-parameter falnily of fimctions, differential forms or 
vector fields on M is assumed to be re9ular in t, i.e., depending on t mmlytically 
(in the C  category) or smoothly (in the C  category). 
Proposition 3.1.b. Proposition 3.1.a holds with the forms 
and function f replaced by familles ut, #t, t't, ff, t  [0, 1], provided that in the 
odd-dimensional case either the set S = {H = 0} bas the eztension property (sec 
Section 1) or the familles ut and t't depend on t polynomially. 
This proposition is also proved in Appendix A, using the already lnentioned 
general theorem on division properties. 
Remark. Proposition 3.1 also holds for germs at a fixed point. 
Another implication of condition (A) concerns the structure of the set Sing(L) 
of singular points of the characteristic foliation L: it cannot be too degenerate. 



2868 B. .]AKUBCZYK AND M ZHITOMIRSKII 

Proposition 3.2. If a Pfaff equation (w) satisfles coditio (A) and in the odd- 
dimensioal case the Martinet ideal of (w) bas the propert9 of zeros, then any 
characteristic vector fietd X of(w), the Martinet hypersurface S and the set Sing( L ) 
of singular points o.f the characteristic foliation bave the followin 9 properties. 
(i) If n, = 2k, then the set M \ Si9(L) of points where X does hot vanish (i.e. 
() is quasi-cottact) is dense in M'. 
(il) If  = 2h + 1. then the set iII  \ S (i.e., the set of points af which  is 
contact) is demse in M'. Equivalently, any 9enerator H of the Martinet ideal is 
ot a zero divisor. 
(iii) If  = 2k + 1. the the set S \ Si9(L) is dense in S. 
Pro@ Statelnent (i) follows froln thc observation that if a is liOt quasi-contact 
at ails" point of ali open set, then any charactelistic vector field X vanishes on 
this set. (vanishes on a connected COlnponent of M', in the analvtic category). 
Consequently, givell a point p in this set. the charactcristic ideal Ip at p generated 
by the çoetïficielits of X is trivial and contains no llOn-zero-divisor. This COlitradicts 
assmnl)tion (A). 
Statolnent (ii) is a simt)le inq)lication of the property of zeros of the Martinet 
ideal: see Definition 1.5 raid the exami)les following it. 
To prove (iii), assulne that there exists a neighbourhood U in 3I" of a point p G S 
such tliat a characterislic vector field X vanishes at any point of the set U ç/S. Bv 
the property of zeros of tlie Martillet ideal we obtain that Xp = 0 nlod(Hp), where 
thc subscript indicates the gerln at p. This contradicts assulnption (A) at the point 
p. The proof is COlnplete. [] 
Propositions 3.1 and 3.'2- inlply the possibility of choosing t he Saille characteristic 
vector field for all Pfaff equations with the saine characteristic foliation. 
Proposition 3.3. Let Pt = (--'t), t  [0.1], be a famil9 of Pfaff equations on 
Iii ' satisfyin 9 the assumptions of Theorem 1.1, if » = 2h, or of Theorem 1.3. if 
•  = 2k + 1. Then for any family Xt of characteristic vector fields of (wt) we bave 
Xt = RXo, if  = 21,', or 
Xt = R Xo nlod (H), if  = 2k + 1, 
where Rt, t  [0.1], is a family of positive-valued fumtions and Xo i.s Xt with t = O. 
Pro@ Let n = 2],'. Tlie equality L(at) = L(a0) inlplies that (X, A Xo)(p) = 0 for 
ail p G M'. By Proposition 3.1.b we obtain tliat X = RXo, where R is a family 
of functions. Proposition 3.1.b also ilnplies that for any fixed t we liave X0 = QX, 
where Qt is a fimction on _M . This leads to the relation (1 - t?Qt)Xo = O. Bv 
Proposition 3.2. X0 does not vanish on a dense subset of M'; thus RtQ = 1. This 
irai)lies that Rt is a family of nonvanishing fimctions. This falnily is positive value& 
since for any p G/il  the fimction t?t(p) is contiliuous in t and Ro(p) = 1. 
In the case of r = 2k + 1 the equality L(wt) = L(wo) gives (Xt A X0)(p) = 0 
for all p G 5'. From the property of zeros of the Martinet ideal v«e deduce that 
Xt A X0 = 0mod(H). Using Proposition 3.1.b, we see that Xt = t?Xomod(H), 
fol" a fainily of fimctions Rt. Similarly, we have X0 = QtXt mod(H) fol" any fixed t, 
where the Q are filnctions. Therefore, (1 - t?Qt)Xo = 0inod(H). Bv Proposition 
3.2, I?Qt = 1 on S, and so/7 is nolvanishing on 5'. From the fact that Rt(p) is 
continuous in t and ri'oin t?o(p) = 1. we deduce that Re is positive vahled on S, 



DISTRIBUTIONS OF CORANK 1 2869 

for anv t E [0, 1]. Finally, adding to Rt the flnction ('l-I 2 with a sufficiently large 
constant C, we obtain Rt positive valued on/I. [] 

Condition (A) implies one lnore division property that xve need in our proofs. 
Its proof is postponed to Section 4. 

Proposition 3.4. Let Pt = (wt). t ff [0.1], be o fo,nily of Pfaff eqaatio, on 3I 2k 
satisfying the ass'umt)lio'ns of Theorem 1. l, and let fit be a family of 1-forms such 
that wt A Pt = 0. Then dt = h.twt .for some family ht of fu,wtions. 

Note that this sta.tement is trivial if wt is a falnilv of lonvanishing 1-forlns. but 
we do hot assume this in our theorems. 

4. A('XILIARY ALGEBRA1C LEMMAS 

To prove the solvability of the homot(q)y equation in the pro(frs of TheorellS 1.1 
and 1.3. we will also usc the following siml)lç algei)raic facts. 
Rccall that a 1-form e on 3I ', n _> 3. is called contact (quasi-contact) at p Ç 
M' if n = 2/,: + 1 (resl)ectively, n = 2k) and ((t A (da))(p) # 0 (respectively, 
(c A (d(@-l)(p) ¢ 0). 

Lemma 4.1. Let c aztd A be 1-forms on M 2. ff c is quasi-contact at p and 
(,x / o/ (do) -2) () = ,I. te ( / et)() = O. 

Lelnlna 4.2. Let o be a l-form on/i2k+1. If ( is a contact al p and A is a 1-form 
such that (A A et A (d(Q k-l) (p) = 0 and (A A (da) k) (p) = O. then A(p) = O. 

The facts stated in these lelmlmS are invariant with respect to multiplication of 
a by a nonvmfishilg function, i.e., they are properties of the Pfaff equation (c). 
These properties Call 1)e easily checked in the Dari)oux coordinates in which the 
Pfaff equation takes the forlu (dz + .rldyl +"- + .r,.dy), where r = 
and r=kifn =2/,:+1. 

Lemma 4.3. Let c be a l-.form on ri/2k+l that is hot contact at p, but c(p) ¢ O. 
If A is a 1-form such that (A A a A (do)-)(p) = O. then (A A (da))(p) = O. 

Pro@ We take a nonzero vector v Ç Tpdl 2k+l such that t, Jct = vida = 0. (The 
existence of such a vector follows ri'oto the assumption that o is not contact at p.) 
Then the relation assunled iii the lelmna implies that the form (viA) • c A (dc@ -1 
valfishes at p. It follows that if (o A (da)-)(p) ¢ I). then (vJA)(p) = 0. md 
consequently ( A (da)k)(p) = 0. On the other hand. if (a A (da)t'-1)(p) = 0. then 
the assumption c(p) ¢ 0 implies that (da)(p) = 0, and then again (AA(da)k)(p) = 
0. [] 

Lemma 4.4. /f ft is a volume form on Al ', A is a 1-form, 7 is an (n - 2)-form 
and X is a vector field defined by the relation X J f = A A 7, then X ] A = O. 

Pro@ To prove this statelnent, note that the definition of X ilnplies X J (AAT) = 0, 
and consequently (X ]A) -7-t- (X ]7) AA =0. It follows that (X ] )" (AAT)=0. 
Since X valfishes exactly at points at which the form A A7 vanishes, we obtain that 
xii=o. [] 

Finally, we need the following general properties of a characteristic vector field. 



2870 B. JAKUBCZYK AND M. ZHITOMIRSKII 

Lemma 4.5. If X zs a characteristic vector field of a Pfaff equatzon (w) on 1I 2k, 
tl en 
X]w = 0 and (X]d,z) Aw = 0. 
Pro@ The first relation follows ri'oto the definition of X and Lemma 4.4. The 
definition of X implies that X] (w A (dw) '-) = 0, which, together with X]' = 0, 
gives (XJdw) A w A (dw) k- = 0 if k > 1. Now the second relation in Lemma 4.5 
follows froln Lemma 4.1 at points where w is quasi-contact. At all other points the 
field X vanishes, and thcre is nothiug to prove.  
Lemlna 4.6. [f X is a characteristic vector field of a Pfaff eqatio (w) on M 2+1 
wlose Martinet ideal bas the propertg of zeros, and H is a generator of this ideal. 
then 
XJw=0 mod (H), XJdH=O mod (H), (XJdw) Aw=0 mod (H). 
Pro@ Due to the I)roperty of zeros of (H), it suffices to prove the three relations 
at any point p of the Martinet hypersurface S such that X(p)  O. The relation 
(XJdH)(p) = 0 follows immediately from the definition of X. To see the other two 
relations, note that S is regular in a neighbourhood of a point p such that X(p)  O. 
The definition of the characteristic vector fiel(1 X in the case n = 2k+ 1 implies that 
the vector field X]s on S is, in a neighbom'hood of such a point p, a characteristic 
vector field of the Pfaff equation (w]s) on S (which is qui-contact at p). Thus 
the remaining two relations follow from Lemma 4.5. The proof is complete.  
Proof of Proposition 3.4. Let Xt be the characteristic vect.or field of (wt) defined 
bv XtJ = wt A (dwt) -. Since Xt (and so wt) does not vanish on a dense sub- 
set of al , the condition t A/t = 0 and Lemma 4.5 imply that Xt]flt = 0 and 
Xt] (fit A (dwt) k-I) = 0. From Proposition 3.3 we have the equality Xt = RtXo, 
with Rt nonvanishing; thus X0J (fit A (dwt) k-l) = 0. Therefore the division prop- 
erty in Proposition 3.1.b implies the following relation: fit A (dt) k- = No ] Pt = 
(gt/Rt)Xt ] , where  is a volume form, gt is a family of flmctions and t = gt. 
Taking ht = gt/R, we tan rewrite this relation iii the form 
(t - brut) A (dt) - = O. 
Let us show that this relation implies fit- tt = O.  know that (fit- htwt)Awt = 
0, since fit A wt = 0. Fix t and a point p at which wt is qui-contact. At this point 
wt does hot vanish: therefore (fit - btwt)(p) = rwt(p), with the scalar r depending 
on t and p. Then the displayed relation implies that r(wt A (dwt)k-)(p) = 0 and 
consequently r = 0. So, (fit - htwt)(p) = 0 if p is a point at which wt is quasi- 
contact. By Proposition 3.2. (i) the set of such points is dense, and so fit = htt at 
any point of the manifold. The proof is complete.  
Now we are ready to prove the solvability of the homotopy equation (HE) and 
out main theorems. 

5. PROOF OF THEOREM 1.1 
Solvability of the homotopy equation (HE) in Section 1 is equivalent to solvability 
of 
( dwt) )k-2 
(5.1) Lz, a.'t ÷ -- Awt A (dw't =0, 



DISTIRIBUTIONS OF COIRANK 1 2871 

with respect to a fmnily Zt of vector fields. NaInely, equation (5.1) is obtained from 
the homotopy equaltion by extlernal multiplication by wt A (dt) -2. Cnversely, if 
(5.1) is solvable then, using the fact lhal the set of quasi-contacl points of (wt) is 
dense in BI" (Proposition 3.2.(i)), we get from (5.1) by Lelnma 4.1 that (Lztwt + 
(dwt/dl))Awt = 0. Therefore, by Proposition 3.4 we get Lz, wt +(dt/dt)+htt = 0. 
for a fmnily of fllnctions ht, which is the holnotopy equat.ion (HE). 
A solution Z, of equation (5.1) will be constructed within the set of fmnilies Zt 
satisfying 
(.2) z, ] , = 0. 
Condition (5.2) ilnplies that Lz, wt = Zt ] dwt, and the equation (5.1) tan be 
rewritten in the forln 
dt )k- 
(5.3) Z](wtA(dt)-l)+('-l)AwtA(d, =0. 
In order to solve equation (5.3} we fix a vohlnm form tl and define a familv Xt of 
characterist.ic vector fields of (wt) by the relaion Xt ] Q = wt A (dwt) k-l. Lemlna 
4.5 and Proposition 3.3 imply the relations XoJwt = 0, XoJ(dwt/dt) = 0 and 
(Xo]dwt) A wt = 0. Thus 
X0]ut = 0. where oet =  Awt A (dt) - 
Therefore, by the division property in Proposition 3.1.b, we ham 
with solne Nlnily fit of (n - 1)-forms of the saine regularity with respect to t as in 
wt. Using Proposition a.a again, we obtain 
for some, regular in t, family of ( -- l)-forms pe. This relation allows to rewrite 
equation (5.3) in the form 
&](X,] )+(-I)X,],=0. 
The latter equation bas a solution Z« defined by the relation 
It now renmins to check that the construçted solution Z satises relation (5.2). 
The equality (5.2) is equivalent to the relation  A   0. From (5.5) and the 
definition of oe we bave (X ] ) A   0. By Lemma 4.5. XJwt  0; therefore 
X ] ( A )  0. Thus the n-form  A « vanishes at any point at which X« 
does not vanish. By Proposition 3.2, (i) the set of such points is evervwhere dense; 
therefore p« A   0 and (5.2) holds. This completes the proof of Theorem l.l. 

6. PROOF OF THEOREM 1.2 
We will use the following proposition (its proof is postponed to the end of this 
section). 
Proposition 6.1. Assume that Po = (w0) satisfles condition (A) and 
(6.1) w/ (d) k-1 = w0/ (da0) k-l. 



2872 B. JAKUBCZYK AND M ZHITONIIIRSKII 

Tbe for the patb  = (1 - t)wo + t we bave 
(6.2) cor A (dct) k-1 = A t 0 A (d0) k-l, 
where Af is a family of functions, polynomial i t. 
Proof of Theorem 1.2. The equality L(0) = L() implies X0 A X = 0. where 
Xo and X are characteristic vector fidds of P0 and P1- Condition (A) satisfied 
for (0) enables us to use the second division property in Proposition 3.1.a, (i) to 
dedu«e that X = RXo and. equivalently, 
(6.3)  A (dWl) k-1 = R0 A (dw0) k-l, 
where R is a Slnooth or analvtic flmction. In fact, R is positive value& which wil] 
follow ffoto the closeness of wa to w0- Therefore. assunfing R > 0, we choose the 

gellerator 

and we have 
(6.4) 
Let 
(6.5) 

£1 / (d&l) k-1 = '0 / (dw'o) k-l- 

t = (1 - t)co0 +/I- 
To prove Theolen 1.2 it is sufficient to show that the family of Pfa.ff equations 
satisfies the assumptiols of Theorem 1.1. 
The equality (6.4) allows us to use Proposition 6.1 to conclude that the relation 
(6.2) holds for the path (6.5). It is «lear that (6.2) implies that the familv (wt) 
satisfies the assmnptions (a) and (b) of Theorem 1.1 provided that the fun«tions 
At in (6.2), t G [0, 1], vanish at no point of 3I . This will follow froln the assump- 
tion on the C-closeness of wt t.o 0 and Theoreln B1 in Appendix B. Define a 
characteristic vector field Xt of (wt) bv the relation XtJQ = t A (&t) -1. where 
Q is a vohune forln, By (6.2) we bave Xt = AtX0. The C-closeness of '1 O '0 
implies the C-closeness of Xt, t Ç [0, 1], to X0. Bv Theol'eln B1 the C-closelmSS 
of Xt to X0 in the equality Xt = AtXo ilnplies that the function At, t  [0.1], is 
C-«lose to 1. ÇOlsequently, Af valfishes at no point of the lnanifold. The proof 
of Theorem 1.2 is COml)lete. 
Proof of Proposition 6.1. Using (6.1), we may assume that the chara«teristi« vector 
fields Xo and X of (#0) and (#), respectively, are equal.  shall prove that 
(6.6) X0 J (w, A (dwt) k-l) =0. 
Having (6.6), we can use assumpt.iox (A) and the division property in Proposition 
3.1.b (wit h polynolnial dependence in t), which gives wt A (dwt) k-  = XoJ Pt, where 
pi is a volume form. Let 'o A (d#o)  = Xo].  = A. Then we get (6.2). 
To pi'ove (6.6), we note that bv Lelmna 4.5 we have 
(.7) XoJ,o = xOJ«l = 0; 
therefore XoJwt = 0. It follows that in order to prove (6.6) it suffices to prove the 
equality 
It is enough to prove the equality (6.8) at any point p such that Xo(p) ¢ O. At 
such a point o(P)  0 and, since Xo = X1, (p)  O. From Lelnlna 4.5 we have 



DISTIRIBUTIONS OF COIRANK 1 2873 

(XoJdwi) A wi = 0, i = 0, 1. Thus, there are flulctions h0 and hl, defined in a 
lieighboul'hoo«l of p, sUC|l that iii this imighbourhood ve bave 
(6.9) XoJdwo = howo. XoJdw = hll. 
XX will prove below that 
(6.10) h t = h. 
Thell frolll (6.9) we get XoJdwt = hwt, where h = h0 = bi, and so (6.8) holds. 
We will show that (6.10) follows ffoto (6.1}.  take thc Lie derivative of both 
parts in (6.1) along the veçtor field X0. Using thc forlnula Lxq = d(XJ q) + XJdq 
for the Lie derivative, we obtain 
= 0 + k(X0Jdw0) A (da0) -1 = t'haa0 A (dwo) -1, 
and siinilally 
L.,- (1  (dl) -') = t,,l  (1) *-1. 
Conq)al'ing these equalities and using (6.1) again, we gel the required relation (6.10} 
(since w0 A (dwo) t- = w A (dw) t-I does llOt vanish on a dense subsçt of 3I). 
Proposition 6.1 is proved.  

7. PIROOF OF T.HEOREM 1.3 
Since the lklartiiml hypersurfaces of Pt = (wt) are the samc for all t. the lklartinet 
ideals are the Saille bv t.]le I)roI)erty of zeros. Thus we Call fiX a generator H of these 
ideals. The following two propositions will hold ulder the assumptions of Theorenl 
1.3. In the propositions ail fanlilies are regular with respect to t (slnooth in the 
C œ category and almlytic iii the (7" category). 
Proposition 7.1. There ezists a familg of vector fields l't satisfging the relation 
( 
(7.1) Ly, w, + dt J Awt A (dw,) k-1 =0 mod (H). 
Proposition 7.2. Let pt be a familg of 1-forrns such that 
(7.2) tzt A Wt A (dcdt) k-1 = 0 niod (H). 
Then the equation 
(7.3) Lztwt 
bas a solution ( Zt, ht ) . 
The solvability of the honiotopy equatioli (HE) in Section 1 is a direct corollary 
of these propositious. Nalnely, we take ttt = --Lytcd t -- dwt/dt, and then the pair 
(2t, ht), with 2t = Zt + t, solves the holnotopy equation (HE) 
Proof of Proposition 7.1. We fix a vohlme form t and define a falnily Xt of char- 
acteristic vector fields of (wt) by the relation XtJf = wt A (dwt) k-1 A dH. Frolll 
Proposition 3.3 we have 
X t = tt.\" 0 lllod (H), 
where/t is a familv of nonvmiishing fllllctiollS, regular in t (of the saille regularity 
in t as in the falnitv wt). Bv Lenima 4.6 we bave XtJwt = 0 mod (H) and 
{XtJdwt) A wt = 0111od (H). V'P Illay repla.ce X with Xo iii these equalities, hl 



2874 B JAKUBCZYK AND M. ZHITOMIRSKII 

particular, we get X0]wt = 01nod (/-/), which ilnplies Xo](dwt/dt) = [Imod (H). 
Taking ail these equalities into account, we see that 
(7.4) X0]  Awt A(dwt) -1 = 0 mod (H). 
This equality and Proposition 3.1.b inlply that 
dwt 
d A wt A (dwt) k-1 = Xo J (f,) mod (H), 
where ft is a falnily of fimctions, regular in t. Replacing X0 with RÇ1,Yt and using 
the definition of Xt, we see that we can rewrite this relation in the forln 
 A wt A (dwt) k-1 
dt 
where gt = ft/Rt. This allows us to rewrite equation (7.1) in the form 
(7.5) (L,wt - gtdH) A wt A (dwt) -1 = 0 mod (H). 
It is clear that (7.5) hohls if  satisfies thc relations 
(7.6) l) J dwt = O, kt J wt = gtH. 
since in this case 
Lwt = d(})J w) = d(gtH) = gtdH mod (H). 
Since (H) is the Maltinet idem of (wt), we have 
(7.7) 
for a falnily St of nonvanishing functions which has the sanie regularity in t  in 
wt (this follows froln the regularity of the leh-hmld side and the fact that division 
by H is a continuous linear operator in the space of Slnooth fnllctions, see Theorem 
B1 in Appendix B). Let us show that (7.6) holds for the family }) defined by 
(7.s)  ] = (d) . 
In fact, applying }] to (7.8), we get gt(}Jdt)A(dwt) - = 0. This relation ilnpli 
 J dwt = 0 (at points where g,(p) = 0 we have }(p) = 0, and at other points we 
can use Lelnnla 4.2 with A = (]dt) and the fact that contact points are dense). 
We have shown the first equality in (7.6). hl order to prove the second one we 
apply })] to (7.7) and, using (7.8). we obtain that (}] wt- gtH)" (dwt)  = 0. This 
implies that ) ] wt - gtH = 0 at points where the form (dt) k does not vanish, in 
particular, at points where wt is contact. By Proposition 3.2, (ii) the set of such 
points is everywhere dense; therefore }]wt -gtH = 0 everywhere, and so (7.6) 
holds. Proposition 7.1 is proved. 
Proof of Proposition 7.2. Bv Lemma 4.2, (i) and the fact that the set of contact 
points is dense in BI  (Proposition 3.2. (ii)), the equation (7.3) reduces to the 
following tvo equations: 
(7.9) (z,) ,  (d,) - = ,, '  (d.,) «, 

(7.10) (Lz, w, + h,,) / (d,,) k = ,, / (d,,)  
(with unknown Zt and ht), obtained from (7.3) by external multiplication by the 
forlns wt/ (dw,) -1 and (dwt) , respectively. 



DISTRIBUTIONS OF CORANK 1 2875 

To solve equation (7.9) we use assmnption (7.2). By this assumption 
(7.11) Pt A cor A (dwt) k- = Hut 
for some familv u of 2k-forms, regular in t })y Theorem B1 in Appendix B. This 
permits us to find an explicit solution Z of (7.9). Namely, since H is a generator of 
the Martinet idem of (w), we bave relation (7.7), i.e., w A (dw)  = H&[, where 
& is a fmnily of nonvanishing flmctions, regular in t. Let ris show that the family 
of vector fields Z defined by the relation 
k 
is a solution o[ thc equation (7.9). e]at.ion (7.1]) and the fact that is hot a zem 
divisor imply that ut A wt = 0. This and (7.12) imply that 
z, ] , = (. 
Consequently, Lz, wt = Zt ] dwt and 
(r.la) (Lz, co,)co,(d,) -' = (Z, J d,),(a,) -1 = Z, J (#,  (d,/). 
k 
Now (7.9) follows from equalities (7.7) and (7.11)-(7.13). 
To prove Proposition 7.2 it renmins to solve equation (7.10) with respect to 
Since Zt]wt = 0. then (Lz,t) A (dt)  = (k + 1)-Zt](dt) TM = 0, and the 
equation (7.10) takes the form 
htwt  (d,)  = vt  (d#t) . 
Due to relation (7.7), to prove that this equation has a solution ht it suces to 
prove that #t A (dwt)  = HCt, where Ci is a family of fimctions, regular in t. 
shM1 first prove that 
(r.14) (,(d,) )(v)=0. f VeS- 
This follows from relation (7.2). Namely, since t is hot contact at a point p  S. 
thus (7.14) follows from (7.2) bv Lenmm 4.3. provided that wt(p) ¢ 0. Since the 
set of points of S at which wt vanishes is a subset of the set Sig(L), the set of 
points p  S where wt(P) ¢ 0 is dense in S by Proposition 3.2. (iii). Therefore 
(7.14) holds at ail points p  S. By the property of zeros of the ideal (H) we obtain 
t A (&t)  = HCtiL Since t and t are regular in t, we deduce from Theorem B1 
in Appendix B that Ct is regular in t. Proposition 7.2 is proved. 

We have completed the proof of Theorem 1.3. Note that the extension property 
of S was used only when referring to Proposition 3.1.b., and therefore it is hot 
needed if cor is polynomial in t (cf. the remark after Theorem 1.3). 

8. PROOF OF THEOREM 1.4 

Since the Martinet hypersurfaces for Po = (oz0) and P = (COl) are the same and 
the Martinet ideals have the property of zeros, they are equal and we tan choose 
a common generator H which will be used throughout the proofs. The following 
proposition holds under the assumptions of Theorem 1.4 and will enable us to 
reduce the problem to Theorem 1.3. 



2876 B JAKUBCZYK ANDM ZHITOMIRSKII 

Proposition 8.1. Assume that 
OE1 / (doE1) k-1 / dH = wo A (dwo) k-1 A dH 
Ttem for the potl t = (1 - t)wo + hZl we Iave 
Wt A (dwt) k-1 A dH = Bt wo A (dwo) k-1 A dH 

(8.1) 
(8.2) 

mod (H) 

mod (H), 

SO 1 A(doel) k-1 A dH = co0 A (dw0) k-1 A dH mod (H). 

cor = (1 -- t ) wO q-tdOl. 
To prove Theoreln 1.4 it is enough to show that the family of Pfaff equations (wt) 
satisfies the assuml)tions of Theorem 1.3. Note that we do hot need the extension 
property of ,5'. since the familv cor in (8.6) is polynomial (in fact. affine) in t, and 
in this case Theorem 1.3 was I)roved without using this assmnption. 
The equality (8.5) enables us to use Proposition 8.1. h is clear that relations 
(8.1) and (8.2) imply that the falnily (wt) satisfies the assmnptions (a), (b) and 
(c) of Theorem 1.3 provided that the fimctions Bt do hot vanish on S (then the 
characteristic line field does hot change) and Ct do hot vanish on 3I ' (then the 
Martinet ideal does hot change), for t  [0, 1]. 
The fact that Bt and Ct do hot vanish follows from tlle 
0 and Theorem B1 in Appendix B. Since CO 1 is C-close 
field Xt is C close to X0. The relation X = /?X0 mod 
the inverse to the operator Lx.H in Theorem B1 imply that 
/) that is C-close to 1 and equal to  at anv point of the 
S. Bv the property of zeros of the Martinet ideM, /) = 
replace  bv / in (8.3) and in the definition (8.4) of &l- 
CC-close to CO0- Define a falnily of characteristic vector fields Xt bv the relation 
XtJçt -= COr A (dCOt) k-1 / dH. Then Xt,t  [0, 1], is Coe-close to X0. The equality 
(8.1) is equivalent to Xt = BtXo mod (H). We again use continuitv of the inverse 
to the operator Lx.H in Theorem B1. By this theoreln there exists a fimction /t 
that is C-close to 1 and equal to Bt at any point of S. Therefore Bt > 0 at any 
point of S. 

(8.4) 
and we gct 
(.5) 
Let 
(S.) 

COr A (dcot) k --- Ct CO0 A (dCO0) k, 
wher'e Bt ad Ct are fizmilies of flnctions, polynomial in t. 
Proof of Thcorem 1.4. Let X0 and .k' be characteristic vector fields of P0 and 
P defined via the saine volume tortu and thc saine generator H of the Martinet 
ideal. Since L(wl) = L(wo), then X A X0 = 0mod (H). Froln condition (A) 
and thc division property in Proposition 3.1.a we obtain X = HX mod (H) or, 
equivalent ly, 
(8.3) 1  (dwl) k-1  dH = RWo A (dw0) k-1 A dH mod (H), 
where  is a smooth or analvtic flmction.  will later show. using closeness of w 
to w0. that  is positive vahmd. Thcrc%re, we can change the generator of Pt for 
1 = R/w, 



DIST1RIBUTIONS OF CO1RANK 1 2877 

To prove that Ct vanishes at no points of àl n, we also lise the C°°-closeness of 
cet. t E [0, 1], to w0 showll above. Let wt A (dwt) k = QtHf, where  is a volume 
form. Relation (8.2) iml)lies that Qt = CtQo. The C°-closeness of cet to co0 implies 
the C-closeness of the fimction HQt to HQo. By continuity of the inverse to the 
operator f ---, fil (Theorem BI) we get the C-closeness of Qt to Qo. Since Q0 
is a nonvanishing flmction, tlwn Ct is Coe-close to 1. The proof of Theorem 1.4 is 
complete. [] 

Proof of Propositio 8.1. It is enough to prove the equalities 

-\'o J (cetA(dce) k-1AdH)-----Il mod (/4), 

Namely, eqnality (817) and conditi»n (A) allow us to use the divisim properties in 
Proposition 3.1.b to conchlde that cor A (dcot)/'-1 A dH = Bt.\']t2mod (Il), and 
so the validitv of relation (8. l), where Bi is a famih of fimctions, polynomial in 
t. Equality (8.8) ilnplies that cor A (dcot)  vanishes at those points of S at which 
X0 does hot vanish. By Proposition 3.2, (iii) the set of snçh poiuts is dense in S: 
therefore cor A (dcet)  vanishes at all points of S. By the property of zeros of the 
Martinet ideal we have cet A (dcet)  = 0 mod (H), and consequently (8.2) holds for 
SOlne familv of fnnctions Ct. polynomial in t. 
In order to prove (8.7) and (8.8) we use the assmnption of Proposition 8.1 and 
choose characteristic vector ficlds X0 and XI of (co0) and (coi) eqnal modulo (H). 
By Lelmna 4.6 we have 

.\]w, = 0 mod (H). XoJdH = 0 mod (H). 

for i = 0, 1, and therefore X0Jwt = 0 mod (H). It follows that in order to prove 
(8.7) and (8.8) it suffices to prove the equality 

(819) (Xo]dcet) A cet : 0 mod (H). 

Due to the property of zeros of the Martinet ideal, it suffices to prove this equality 
for any point p Ç S such that Xo(p) 7  O. At snch points the 1-fonn dH does hot 
vanish and S is smooth. Since XI = -\-o mod (H), using Lemma 4.6 we obtain 

(Xo]dceo) A o = (X0Jdce) A co I : } 

in a neighbom-hood of p in S I Since X0(p) ¢ 0. then oto(p) -¢ 0 and cel(P) ¢ 0, and 
therefore these relations imply the equalities 

(8l 10) 

XoJdce0 = hr.cco, XoJdvl --- ]licol, 

which hold iii a neighbourhood U of p in S. Here ]10 and ]il are functions defined in 
this neighbourhood. We will show that ho = hl; then (8.10) implies that Xo]dcet = 
]10cot, and (8.9) holds in the neighbourhood U. 
To prove that ho = ]il Oll []" C S, we restrict the relation assmned in Proposition 
8.1 to the tangent bundle of . We obtain (coi A dcel)[u = (co0 A (dco0)k-)l and 
take the Lie derivative of this relation along the restriction Xols of X0 to S (recall 
that X0 is tangent to S). As iii the proof of Proposition 6.1, we obtain the required 
equality ]10 = bi. Proposition 8.1 is proved. [] 



2878 B. JAKUBCZYK AND M. ZHITOMIRSKII 

APPENDIX N. DIVISION PROPERTIES 
In this Appendix we present a general theorem on division properties of the 
exterior (respectively, interior) product with a section X of a vector bundle. This 
theorem is proved in [DJ] and implies our Propositions 3.1.a and 3.1.b. Our results 
hold in the categories C s, where s = ex2or s = w. 
Let AI be a paracompact differential manifold. Consider a vector bundle E over 
'I of tank m and denote by E* its dual bmdle. Let Af = Af(E) denote the rth 
exterior power of E, r = 0, 1 ..... m, with A0 =/I x  and Ai = E. We denote , 
A(M: E) the linear space of sections of A,, (smooth or real analytic, depending on 
the category). 
Any section cv of E defines lhe linear operator of exterior multiplication bv c. 
which gives the complex 
(A.I) 0 --, A0(M) ---, A(M: E) --, ...... --, A,,(M: E), 
wflh the operator 0« = 0 : Av(M: E) --, Av+(M: E) defined by 0(')') := o A % 
Consider a section X of the dual bundle E*. This section defines the operator 
of the interior product with A', Ni : A,.(M: E) --, A_I(M:E). Given a local 
basis e .... , e of E, the operator of the interior product with X is defined on the 
elements of a local basis of A bv 
xJ(%  ...  e) = (-1)-' (x. %) «,  ...% ...  e, 
3=1 
where êi means absence of ei mld (-, .) denotes the dualitv product between E* 
and E. Clearly, (X])2 = 0; so the operator A'] defines the complex 
(A.2) 0--, A,(AI: oe)--, A,_a(AI: oe)--, ...- Al(AI:E)-* Ao(M: E). 
Let S be a closed subset of .àl. Denote by A(M. S: E) C A,.(M: E) the subspace 
of sections of Af(M: £) vanishing at ail points of S. and let 
A(S: E)= A,.(AI: E)/A,.(AI, S; E) 
denote the quotielil space. 
Any element  of AI (ç; E) defines the unique operator 
0 : Ap(S: E) --, Ap+a (S: E) 
(the quotient of the operator of exterior multiplication), which gives the complex 
(A.3) 0--, Ao(S)--, A,(S: E)--, ...... --, A,(S: E). 
Given a section X of £*, the operator XJ defines the following complex on the 
quotient spaces: 
(A.4) 0--- Ara(S; E) Am-I(S; E)---.--- A(S: E)-- A0(S: E). 
We define the invariant dp(X) = depth(Ip), where Ip is the idem of function 
germs at p G M generated by the coelïïcients a ..... a of X in a local basis of E* 
(cf. Definition 1.2 in Section 1). Sinfilarly, given a pair (H, X) of a flnction H and 
a section X of E* on M, we define dp(H. X) as the maximal length of a regular 
sequence of function germs that begins with the germ Hp of H at p and has further 
elements in In (cf. Definition 1.4). Analogously we define the invariants dp(a) and 
dp(H, 



DISTRIBUTIONS OF CORANK 1 2879 

Statements (i) and (il) of the following theorem hold in the C œ and C  cate- 
gories, for 0 _< q _< n- 1. 
Theorem A. (i) If « satisfles the condition dp(o) >_ q + 1 for all p E .I such 
that c(p) = O. thon the complex (.4.1) is exact up fo Aq(M:E). Similarlg. if 
dp(X) >_ q + 1 for all p  ]il such that X(p) = O. then the complex (.4.2) is exact 
up to Am_q(M; E). 
(il) Let H be a function on ]il such that lhe ideal (H) bas the prop«rty of zeros. 
and let S = {H = f)}. If et is a section of E on M such that (H.() satisfies 
d(H. c) >_ q + 2 for all p  S such that et(p) = O. thon the conplex (.4.3) is exact 
up to Aq(S; E). Similarl,q. if X is a section of E* on M and d(H, X) >_ q + 2 for 
all p  S such that X(p) = O. then the cornplex (.4.4) is e.ract q fo A,_q(,ç; E). 
(iii) If the assumptions of (i) hold, then. in the C  category, the cornplex (A. 1) 

splits up to Aq-l(]il;/) and 
assumptions of (il) hold and 
splits up fo Aq-l(S;/) and 

t/Le complex (.4.2) split. up to Am-q+] (]il;/). If the 
S bas the extension propertg, then the conpleï: (A.3) 
the complex (A.) splits up to Am-q+l(S; E). Here 

the correspoding spaces are equipped with the C œ topology and are considered as 
Fréchet spa«es (quotient FfCher spaces). 
Above, a complex 0 -- L, -- .-. -- Lrn-q+l -- Lm-q -- ..- defined by the 
operators Oi : Li -- Li-1 is called exact up to Lm-q if Im0i+l -= ker Oi for i = 
m, m-- 1 ..... m --q, and it splits up to Lin-q+1 if the L are linear topological spaces. 
hnOi are closed subspaces of Li-1 and each Oi : Li -- Li-1 has a continuous right 
inverse tçi defined on hn Oi, for all i = m, m - 1 ..... m - q + 1. 
The above theorem follows rioto The(»rems 2.1 and 2.2 in [DJ]. In the local 
case (of germs) statements (i) and (il) follow from a well-known algebraic result on 
exactness of the Koszul complex, cf. e.g. lE] or [JZh2], A1)pendix 1. 
Proof of Proposition 3.1.a. In the even-dimensional case the first implication fol- 
lows trivially from statement (i) in Theorem A if we take the bundle E equal to the 
cotangent bundle E = T'M, the dual E* = TM, and consider the complex (A.2). 
(A(M, E) is identified with the space of diffcrential r-forms on ]il.) The second 
implication follows analogously from the saine statement by taking E = TM and 
the complex (A.1). 
In the odd-dimensional case the first implication follows in a similar way from 
statement (il) in Theorem A concerning the complex (A.4). This is because the 
property of zeros of (H) allows us to identify the elements of A(S: E) with the 
equivalence classes of differential r-forms modulo (H) (cf. out convention on no- 
tation mod (H) presented after Definition 1.3). The second implication follows 
analogously from statement (il) in Theorem A concerning the complex (A.3). [] 
Proof of Proposition 3.1.b. The existence of pt and ft for any fixed t follows from 
Proposition 3.1.a. We have to show the regularity of these familles in t. We shall 
prove the regularity of ttt (the proof of regularity of ft is analogous). If ut depends 
on t polynomially, then Proposition 3.1 allows us to construct #t polynomial in t, 
and the regularity follows trivially. In the general case out argmnents are different 
for the categories C" and C . 
In t.he C" category we use the following fact: if a sequence al ..... a, of rem 
analytic function germs at p  ]il is regular in the ring of analytic function germs 
at p, then it is regular when considered as a sequence in the ring of real analytic 
function germs of the variables (x, t)  M × 11¢ at (p, to), for anv t0 6 [0, 1]. Using 



2880 B. JAKUBCZYK AND M. ZHITOMIRSKII 

local coordinates, this fact can be easily proved for the case of formal power series 
using the definition of regular sequcnce in the ring of fornml power series of the 
variables x ..... xn, t. Then, using the fact that the ring of formal power series 
is faithfully fiat. over the ring of convergent series (see Malgrange [Mlg], Chapter 
3), we see that it also holds for converging series and so for germs of analytic 
functions. Using the above fact we see that the assumption (A) holds over the 
manifold ,ÇI = M x I, where I is an open interval containing [0, 1] on which the 
analytic family vt is well defined by analytic extension. Thus we Call lise Theorem 
A over the manifold .). i.e., for the bundles E = T*M and E* = T3I pulled back 
to IÇI by the canonical projection M x I  21I. 
In the (' category, in the even-dimelsional case the smooth dependence of 
pt on t follows rioto statement (iii) in Theorem A. By this statement there exists 
a continuons right inverse operator K to the linear operator X] : A,.+(M: E)  
A,.(M: E), for r= n-1 and r = n-2, and we can define pt = Kut. Here E = T*M 
and A,.(M: E) = A(M). the space of diffcrential r-tbrms on M. 
In the Coe category, iii the odd-dilnensional case we also use statement (iii) of 
Theorem A and the extension property of S. Namely, fbr E = T*3I we define 
lt = A Kut]s, where Iç : A,-I(S; E)  A(S: E) is the continuous right inverse 
p,'t,r to xJ : .,,(s; )  &,_(s; ). ,,d  : .%(5'; )  .,,(M: ) is  
contimums linear operator of extension.  

APPENDIX B. CONTINUITY OF DIVISION 
Contilmity of division in the cases presented below is needed in the main proofs 
and will be proved separately. Let C(.I) and l'ect'(M) be the spaces of smooth 
functions and smooth vector fields on M. with the C topology. Let C(3I, S) 
and lect'(M..ç) be the subspaces of flmctions (vector fields) on 3I vanishing on 
the Martinet hypersurface S c M. The quotient Fréchet spaces 
C(.ç)=C(M)/C(M..ç), Vect(S:TM)= Vect(M)/Vect(M.S) 
can be idcntified with the space of smooth function on S and the space of blnooth 
sections of the tangent bundle T3I restricted to S. rcspectively. 
Given a Pfaff equation on 3I 2 and a characteritic vector ficld X, we consider 
the linear operator 
Lx : C(M) -+ lect(M). Lx(f) = fX. 
For a Pfaff equation on M t+. a characteristic vector field X and a generator H 
of the Martinet idem we consider the linear operators 
/: C(M)  C(M) • ;(I) = 

and 

where [ • ] denotes the equivalence class in the corresponding quotient space. 
Theorem B1. If the characteristic vector field X satisfies codit.io (A ) and in the 
odd-dimesional case the Marrinet ideal (H) has the propert9 of zeros, then each 
of the lmear operotors Lx. Lt4 a, Lx.tî, is bjective, bas closed ima9e, and hos 
codinuous im,erse defined o the image. 



DISTIRIBUTIONS OF COIRANK I 2881 

Pro@ By the Banach open lnapping theorem in Fréchet spaces it suffices to prove 
that each of the operators L,\- LH, and Lx. is injective and has closed image. 
The injectivity of the operators Lx, LH a.nd L.\-. follows from Proposition 3.2. 
(i), (il), (iii), respectively. The closedness of the image of the operator L follows 
h'om the global property of zeros inlplied bv Definition 1.5 by this property the 
image of LH coincides with the closed subspace C(M, S) c C(M) of hmctions 
vanishing on S. 
To prove the closedness of the image of the operators Lx and L,\-., we use 
Proposition 3.1. By Proposition 3.1. (i) the inmge of Lx coincides with the kernel 
of the continuous operator Vect(M) --+ V-'(A"-TM) defined bv  -- X A]t, where 
F(A2TM) is the space of smooth scctions of the skew-symmetric product of the 
tangent bundle TM, with the C œ topology. Similarly, hv Proposition 3.1, (il) the 
inmge of Lx.n coincidcs with the kernel of the continuous operator l'ect'(S) -- 
F(A2(S:TM)) given by [Y] --+ IX A }'], where F(A"-(S:T3I)) is the space of 
smooth sections over ,_q of the skew-swmnctric pro(hlct of the t.angcnt bundle T3I 
(with the C œ topotogy) and [ • ] denotes the equivalence çlass in the corresponding 
quotient space. Thc kernet of this op('rator is a closod subspace of l'cet ' (S: TIII) = 
Vectoe(l)/l'ectoe(M, S). The proof is complete. [] 

It is natural to ask if it is possible to replace the Coe-closeness of ,Zl to w0 in 
Theorems 1.2 and 1.4 by C-closeness with some r. Any attempt ai answering 
this question requires nlodification of Theorem B1, which was used in the proofs 
of Theorems 1.2 and 1.4. Proving Theorem 1.2. we had to show that the function 
AI. t  [0, 1], does hot vanish at any point of M. In the proof of Theorem 1.4 we 
had to show that the functions BI, I  [0, 1], do hot vanish ai points of S and the 
hmctions Ct, t  [0.1], do hot vanish ai points of 31. The C-ctoseness of 1 to d 0 
given as an sumption in Theorems 1.2 and 1.4 and the continuity of the inverse 
to the operators Lx, Ln and Lx.n allowed us to obtain the C-closeness of AI, 
BI and C, to 1. Of course, to «onclude that these hmctions do hot vanish, their 
C°-closeness to 1 woutd be enough. 
ç introduce the following topological chara«teristic of a linear injective operator 
L " C(M)  COe(M) or L "COe(M)  V««t(M) or L " C(S)  Vect(S). 
Denote by m  {0, 1.2 .... ;OE} the minimal m such that for anv s  0 the con- 
vergence to 0 of the sequence of sections L(f) in the C + topology implies the 
convergence to 0 of the sequence of hmctions f, in the C  topology. This means 
that the inverse to L behaves hot worse than a linear diffcrential operator of order 
m. Note that bv Theorem B1 we have m(Lx), m(L), and ,n(Lx.) 
In many cases the nmnbers m(Lx), m(LH) and m(Lx.H) are finite and can be 
found or estimated from above, sec examples below. Tracing the construction of 
the functions AI, BI and Ci in the proofs of Theorems 1.2 and 1.4. it is easy to 
check that if these numbers are finite, then: 
1. the C-closeness of AI to 1 holds provided that the 1-form 1 i8 close to 0 
in the C  topology with r = 2m(Lx) + 2: 
2. the C°-closeness of BI to 1 holds provided that the 1-form 
in the C  topology with r = 2m(Lx.H) + 2: 
3. the C°-closeness of Ci to 1 holds provided that the 1-form d 1 i8 close to '0 
in the C  topology with r = m(Lx,H) + m(LH) + 2. 
Therefore in Theorems 1.2 alld 1.4 the Coe-closeness of W'l to 0 can be replaced 
by the closeness in a weaker topology, and we obtain the following result. 



2882 B. JAKUBCZYK AND M. ZHITOMIISKII 

Theorem B2. In Theorem 1.2 the C°-closeness of wl to wo can be replaced by 
the CT-closeness with r = 2m(L¥) + 2. In Theorem 1.4 the Coe-closeness of wl to 
wo can be replaced by the CT-closeness with 
r = lnax (2n(Lx.H) + 2. m(Lx.H) + n(LH) + 2). 
Examples (n = 2k). 1. If X has no singular points, then it is clear that m(L_¥) = 
0. Therefore the Coe-closeuess of w to w0 in Theorem 1.2 can be replaced by 
C2-closeness. Wc obta.in Theoreln 0.1. 
2. If the 1-jet of X vanishes at no points of the nmnifold, then it is easy to prove 
that m(Lx) _< 1. Therefore the Coe-closeness of w to w0 in Theorem 1.2 can be 
replaced tri" C4-closeness. 
Examples (n = 2/,: + 1). 1. If (co0) is a Martinet distribution, i.e.. dH(p) ¢: 0 
and X(p) ¢ 0 for an.v p  .b' = {H = 0}, then it is easy to prove that m(LH) _< 1 
and m(Lx.H) = 0. Therefore the C-closeness of CO 1 to CO0 in Theorem 1.4 can be 
replaced b.v C3-closeness. We obtain Theorem (/.2. 
2. Assume that dH(p) ¢= 0 for any p  S -- {H = 0}. Then the restriction of 
X t.o S is a. smooth vector field X]s on S. Assume that the 1-jet of Xs does hot 
vanish. In this case m(L) < 1 and m(Lx.H) <_ 1. Therefore the C-closeness of 
1 to COe in Theorem 1.4 can be replaced by C4-closeness. 

ACKNOWLEDGMENTS 

While working on this paper we have profited fronl discussions with several 
colleagues. We are especially thankful for helpflll advice obtained from Pawet 
Domallski, Jean-Paul Gauthier, Pierre Mihnan and Richard Montgolnery. 

lAI 

[Arll] 

[BS] 
[BH] 
[DJ] 
[E] 
[1 
[JP] 
[JZhI] 

[JZh2] 

t EFERENCES 

A. Agrachev, Methods of Control Theory in Nonholonomic Geometry, Proc. Internat. 
Congress of Math., Ziirich 1994, Vol. 2, pp. 1473-1483, Birkh/iuser. Basel, 1995. Mil 
97f:58051 
V. I. Arnold and Yu. S. Ilyashenko, Ordinary Differential Equations, in Encyclopaedia of 
Math. Sci. Vol. 1, Dynamical Systems 1, Springer-Verlag (1986). Mil 87e:34049; Mil 
89g:58060 
E. Bierstone and G. W. Schwarz, Continuous linear division and extension of C œ functions, 
Duke Math. Journal 50 (1983), 233-271, Mil 86b:32010 
Il. L. Bryant and L. ttsu, Rigidity of integral curves of rank 2 distributions, lnventiones 
Math. 114 (1993), 435-461. MIl 94j:58003 
H. Cartan, Variétés analytiques réelles et variétés analytiques complexes, Bull. Soc. Math. 
de France 85 (1957), 77-99. MIl 20:1339 
P. Domafiski and B. Jakubczyk, Linear continuous division for exterior and interior prod- 
ucts (accepted to Proc. Aner. Math. Soc.). 
D. Eisenbud, Commutative Algebra, Springer-Verlag, New York 1994. MN 97a:13001 
A. Golubev, On the global stability of maximally nonholonomic two-plane fields in four 
dimensions, Internat. Math. Res. Notices, 11,523- 529, 1997. MIl 98g:57043 
J. W. Gray, Some global properties of contact structures, Annals of Math. 69, (1959), 
421-450. Mil 22:3016 
B. Jakubczyk and F. Przytycki, On J. Martinet's conjecture, Bull. Polish Acad. Sci. Math. 
27 (1979), No. 9, 731-735. MIl 82e:58003 
B. Jakubczyk and M. Zhitomirskii, Odd-dimensional Pfafiïan equations; reduction to the 
hypersurface of singular points, Comptes Rendus Acad. Sci. Paris, Série I t. 325 (1997), 
423-428. MR 99e:58003 
B. Jakubczyk and M. Zhitomirskii, Local reduction theorems and invariants for singular 
contact structures, Ann. Inst. Fouiner, 51 (2001). 237-295. MIl 2002e:58001 



DISTIIBUTIONS OF CORANK 1 2883 

[LS] 
[Ilg] 
[Mari 
[NIon] 

[Zhll 

W. Liu and H. Sussmann, Shortest paths for sub-Riemannian metrics on rank-two distri- 
butions, Mem. Amer. Math. Soc., Vol. 118 (1995), No. 564. MR 96c:53061 
13. Malgrange, Ideals of differentiable functions, Oxford University Press, 1966. NIE 
35:3446 
J. Martinet, Sur les singularités des formes différentielles, Ann. Inst. Fouiner, Vol. 20. No.1 
(1970), 95-178. NIE 44:3333 
R. Nlontgomery, A survey on singular curves in sub-Eiemannian geometry, J. Dynamical 
and Control Systems, Vol.1. No.1 (1995), 49-90. NiE 95m:53060 
R. Montgomery and M. Zhitomirskii, Geometric approach to Goursat flags, Annales de 
l'Instztut Henri Poincaré, Analyse Nonlineaire, Vol. 18, No. 4 (2001), 459-493. mie 
2002d:58004 
J. M. Ruiz, The Basic Theory of Power Series, Advanced Lectures in Mathematics, Vieveg, 
Wiesbaden, 1993. NIE 94i:13012 
C. B. Thomas, Y. Eliashberg, and E. Giroux, 3-dimensional contact geometry, in "Con- 
tact and Symplectic Geometry" Publ. Newton lnstitute 8, Cambridge University Press, 
Cmbridge, 1996, 48-65. MR 98b:53026 
M. Zhitomirskii, Singularities and normal forms of odd-dimensional Pfaff equations. Func- 
tional. Anal. Appl., Vol. 23, (1989), No. 1, pp. 59-61. ME 90i:58007 
M. Zhitomirskii, Typical singularities of differential l-forms and Pfaflïan equations, Trans- 
lations of Math. Monographs, Vol. 113, Amer. Math. Soc,, Providence, RI, 1992. MR 
94j:580{14 

INSTITUTE OF NIATHEMATICS. POLISH fl.CADEMY OF SCIENCES. NIADECKICH 8. 00-950 Whrts.w. 
POLAND AND INSTITUTE OF fl.PPLIED lkIATHEMATICS, U'NIVEI:ISITY OF ,VARSAV. POLAND 
E-mail address: B. Jakubczyklimpan. gov. pl 

DEPARTMENT OF NIATHEMATICS, TECHNION, 32(100 IAIFA. ISRAEL 
E-mail address: mzhitechunix, technion, ac. il 



TRANSACTIONS OF THE 
AMER1CAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2885-2903 
S 0002-9947(03)03289-6 
Article electronically published on Match 14, 2003 

WHEN ARE THE TANGENT SPHERE BUNDLES 
OF A RIEMANNIAN MANIFOLD REDUCIBLE? 

E. BOECKX 

ABSTRACT. \e deternfine ail RiemaImian manifolds for xvhich the tangent 
sphere bundles, equipped with the Sasaki metric, are local or global Riemann- 
ian product manifolds. 

1. INTRODI¢CTION 

When studying the geometry of a Riemmmian manifoM (3/. g), it is often useflfl 
to relate it to the properties of its unit tangeut sphere bmMle Tt3I. hl earlier 
work, we have been primarily interested in the geometric properties of T3I when 
equipped with the Sasaki metric gs. This is probat»ly the sinll)lest possible Rie- 
mamfian metric on T3I and it is completely determined by the nletric 9 on the 
base manifold ]lI. In this way, we bave obtained a. nUnlber of interesting charac- 
terizations of specific classes of Riemmmian manifolds. We refer to [2], [5]. [6], [7] 
and the references therein tbr examples of this. Also tangent sphere bundles T,.M 
with radius r different from 1 and equipped with the Sasaki metric bave been stud- 
ied recently ([9], [10]). The geometric properties of these Riemmmian manifolds 
may change with the radius. Sec [9] for an example of this. Of course, other 
tiemannian metrics on the tangent bundle and on the tangent sphere bundles 
are possible. Of these, the Cheeger-Gromoll metric 9cc may be the best known. 
However, for tangent sphere bundles, this specific metric yields nothing new. since 
(T,-M, gcc) is isometric to (T,./4i-4M, gs ). The isometry is given explicitly by 
p: Trl]l -- Tr/ lvq-l[: (X, t) v- (X, u/V + r2). 
It is a.n interesting geometric problem to determine when a tangent sphere lmn- 
dle, which we alwoEvs consider with the Sasaki metric in this paper, is reducible, 
i.e., when it is locallv or glol)ally isometric to a Riema.nnian product manifold. To 
our surprise, we could not find anv results in the literature concerning this ques- 
tion. Nevertheless, knowledge about reducibility could help to deal with geometric 
questions about, tangent sphere bundles, lu [4] for instance, we use it in an essen- 
tial way to determine all unit tangent sphere bundles that are semi-symmetric, i.e., 
for which the curvature tensor at each point is algebraically the saine as that of 
some symmetric space. Actually, that problem was the inspiration for the present 
article. As concerns the local reducibility of tangent sphere bundles, we prove here 
the following. 

Received by the editors November 11, 2002 and, in revised form, January 21. 2003. 
2000 Mathematics Subject Classification. 53B20, 53C12, 53C20. 
Key words and phrases. Tangent spheïe bundle, Sasaki metric, reducibility, Clifford structures, 
foliations. 

@2003 American Mathematical Societ.y 



2886 E. BOECKX 

Local Theorem. A taTgent sphere bmdle (T,.]iI, gs), r > O. of a Riernannian 
manifold (M ', g).  >_ 2, is locally reducible if and mdy tf (M, g) bas a fiat factor. 
i.e.. (lI,g) is locally isometric to a product (M',g') × (llk, g0) where 1 _ k _ n and 
go denotes the standard Euclidean metric on II . 
The corresponding global version reads as follows: 
Global Theorem. Let (]il '.g). n >_ 3. be a Riernannian rnanifold and suppose 
that (TM. gs) is a global Rienmnnian product. Then. (]il, g) is either fiat or it is 
also a global Riemamia product, with a .fiat factor. 
Coversel.q, if(M, g) is a global product space (M', g') × (F , go) where 1 <_ k <_ n 
ad F is a co.nnectcd ad sin@ly connected .fiat space, then (TM, gs) is a global 
Riemannian product, also with (F, go) as a .fiat factor. 
In vicw of thc commcnts above, thcse results remain valid if we consider the 
tangent sphere bundles equipped with the Cheeger-Gromoll metric. 
This article is organized as follows. After giving the necessary definitious and 
formulas concerning tangent sphere lmndles, we show in Section 3 that onlv two 
types of decomposition for TM are possible: a vertical and a diagonal one. The 
special form of the curvature of (TrM, gs) for xrertical vectors is crucial here. In 
particular, the saine procedure does hot go through for the tangent bundle TM. 
Sectiou 4 deals with the diagonal case. We find that a diagonal decomposition 
gives rise to a Clifford representation via specific curvature operators. As a result. 
only base maififolds with dimension 2, 3, 4. 7 or 8 could possibly adinit diagonal 
decolnpositions. The diffelent dimensions are then handled separately. It turns out 
that diagonal decompositious can only be realized for a fiat. surface as base space. 
The general situation with a vertical decolnposition is treated in Section 5 and leads 
to the Local Theoreln above. The final section is devoted to global cousiderations. 

2. TANGENT SPHERE BUNDLES 

We first recall a few of the ba.sic facts and formulas about the tangent sphere bun- 
dles of a 1Riemannian manifold. A more elaborate exposition and further references 
can be found in [51 and [9]. 
The tangent bundle Tll of a Pdelnannian lnanifold (]il. g) consists of pairs (:r, u) 
where 3c is a point in M and u is a tangent vector to M at .r. The mapping 7r : TM -- 
M: (:r,u) - x is the natural projection from TM onto M. It is well known 
that the tangeut space to T]II at a point (a-, u) splits into the direct sure of the 
vertical subspace I'T]II(z,, 0 = kerr.l(,, 0 and the horizontal subspace HT]II(,, 0 
with respect to the Levi-Civita connection 
H T ]I I( ,, 0 • 
For w  T,]II. there exists a unique horizontal vector w   HTM(z., 0 for which 
7r.(w ) = w. It is called the horizontal lift of w to (x, u). There is also a unique 
vertical vector w   I'T]IIt.,O for which w(df) = w(f) for all fimctions f on ]iI. It 
is called the vertical lift of iv to (x, 
and HT]II(,, 0 and VT]II(,,O, respectively. Hence, everv tangent vector to T]II 
at (:r, u) can be written as the sure of a horizontal and a vertical lift of uniquely 
defined tangent vectors to ]Il at :r. The horizontal (respectively vertical) lift of 
a vector field X on M to TM is defined in the saine way bv lifting X pointwise. 
Further. if T is a tensor field of type (1, s) on M and X1,..., X-I are vector fields 
on ]il. then we deuote by T(X1 .....  ..... X_) " the vertical vector field on TM 



WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2887 

which at (ce, w) takes the value T(Xlc,...,lv .... , Xs_ix) v, and sinfilarly for Ihe 
horizontal lift. In general, these are hot the vertical or horizontal lifts of a vector 
field on/ll. 
The Sasaki metric gs on T/iI is completely deIermined by 

9s(x h, ) = 9s(x v. ") = 9(x, ) o , 9s(x . v) = 0 

for vector fields X and }" on/il. 
Our interest lies iu the tangent sphere bundle T,./i1 of some positive radius r, 
which is a hypersurface of Tal consisting of ail tangent vectors lo (/il, g) of lenglh r. 
Il is given implicitly by the equation g,(«, u) = r 2. A unit normal vector field N 
to T/iI is given b.v the vertical vector field "/r. We see that horizontal lifts 
to (:r, u)  TM are tangent fo T/II, bul vertical lifts in general are not. For 
that reason, we define the tangential lift u , of u'  T.Cl to (x, u)  TI by 
1 
wt = wv - 7 g(w.u)N. Clearly, the tangent space lo T5I at (x,u) is spmmed 
bv horizontal and tmgential lifts of taugent veclors to M at x. One defines the 
tangential lift of a vectorfleld X on/iI in the obvious ,va.v. For the sake of notational 
clarity, we will use 2{" as a shorthmd for X -  g(X, u)u. Then .\' = .(v. Further, 
we denole by VT,.kI he (u - 1)-dimensional distribution of vertical tangent vectors 
to TrM. 
If we consider THI with the metric induced from the Sasaki metric 9s of T/i1, 
also denoted by 9s, we turn TrM iuto a Fliemannian manifold. Its Levi-Civita 
Colmection Ç is described completely by 

() 

1 
V x, I "t = o(I",,z)X t, 

Ç\.hY h 

: _ (H%A')y) h, 
2 
1 
= (Vxr)'+((,,.)x) h, 
- (Vx}')h I((X,}'))' 

for vector fields X and Y on/il. It.s Riemann curvature tensor/) is given bv 

(2) 

(X h, Yh)Zt 

(X h, Yh)zh 

for vector fields X, Y and Z on/il. (See [9].) 



2888 E. BOECKX 

3. TWO TYPES OF DECOMPOSITION 
Let (/l.g) be a Fliemmmial mmlifold of dimension , k '2 and suppose that 
its tangent sphere bundle T,.M is (locally) reducible, i.e., (TM, 9s)  (/1,gl) x 
(M2, 9)- A point (.r, u) in TM corresponds to a couple (p. q)  I1 X 2lh, and 
the tangent space T(,,)TkI can be identified with TpI1 @ q.lI2. In the sequel, 
we will write T(,)kI and T(,)kI for TpI1 and TqM2, considered as subspaces 
of T(,)TM. in order not to make the notation too cumbersome. 
Suppose first that, at a point (.r, u) of TkI, the tangent space to one of the 
factors, say to I1, contains a nonzero vertical vector X t, X  TkI and X K u. 
Since we have a Riemannian product, the curvature operator (U, V) preserves 
the tangeut spaces to both factors for ail vectors U and V tangent to TtM. In 
partiçular, it follows that 
1 
(}", xt)x t = r  (9(.¥, X) }-' - g(X. }')X t ) C T(.u)3ll 
for all vcct.ors }  T«;I. As a consequence, I'T5I(.) C T(.u)M, and lI is at 
lcast (,- 1)-dimcnsional. Hencc, (f at a point 4E;l one of the factors contains a 
• nonz«v vertical ve«tor, if «ontains fb« complete vertical distribution al that point. 
 call the decomi)ositiou vertical at (a'. u) in such a situation. Note that this is 
the ce as soon  m{dim fil1, dira M2} > n. Indeed. if dira fill > , then 
- dim(l'E,lt,u ) + 
> (., - 1) +  - (2,, - 1) = 0. 
So, the only possibility %r the decomposition hot to be vertical at (z. u) is that 
dira I1 = , dim.12 =  - 1 {or conversely) and neither factor is tangent to a 
vertical vector. X call this a diagonal decomposition at (x, ). 
The major part of the sequel will be devoted to the diagonal case. Using a purely 
infinitesimal (i.e., pointwise) approach, we show that a diagonal decomposition is 
only possible in one specific situation. Afl.erwards. we study the case of a vertical 
decomposition. 

4. DIAGONAL DECOMPOSITION 

4.1. A suitable basis. In this section, we consider a diagonal decomposition 
Tr]l  1I1 x M.2 at (x, u) with dira/I1 = "n and dira/I2 =  - 1. For dimen- 
sional reasons, we bave 

dim(T(:,,)3ll  HT3I(,.,)) > O. 

Let Xn  TII be a unit vector such that Xn h is tangent to Il[ 1 at (x. u) and 
extend it to an orthonormal basis {X .... ,X,} of T«M. If rc.(,,,)(T(,,,)M1) ¢ 
TM. then there nmst be a vertical vector tangent to/I1 at (x. tt), contrary to the 
hypothesis. Hence, there exist well-defined vectors Y ..... }-t orthogonal to tt 
for which .\-h + tlt ..... Xn_l h ÷ t'n_l t and X, h are tangent to 1[ 1 at (x.u). 
Clearly, they form a basis for T(æ.)M. though uot in general an orthonormal one. 
Moreover. {Y1 ..... t;_, u} is a basis for TM too. Otherwise, there would exist 
a nonzero vector t"  TII, orthogonal to tt and to Yi, i = 1 .....  - 1. But then 
t "t would be orthogonal to X, h and to Xi h + E t, i = 1 .....  - 1. and hence 



WHEN ARE TOENGENT SPHERE BUNDLES REDUCIBLE? 2889 

would belong to (Tla:,u)11Ii) - = T(a:,u)111.2, contrary to the hypothesis that kI2 has 
no vertical tangent vector. 
Next, consider the (n- 1) x (,,- 1) matrix a = (g,(I: ))i4= ..... _. Since 
this nmtrix is symmetric and positive definite, it cm be diagolmlized bv a suitable 
orthogonal transformation: 
p¢pt = diag(A2 .... , kn_l 2) 
where P = (Pij)  0(,- 1) and A, > 0 tbr i = 1 .... , - 1. Ifwe p/lt 
j=l j=l 
fo =  ..... - . thon oth {X5 .... 
or{honormal bases for T,M. Further. thc vectors 
2,  + ,L'=  ,, (a 
logether wi{h Xn  span the {angenl spa«e {o kI at (x. u) and these xwctors are 
pairwise orlhogonal. The tangent space to 
orthogonal vectors 
A,çi -«, i= 1 ..... t-1. 

Finally, we show that all the 
gs(/(U, V)W, T) = 0 at (x, u) 
to/111 and another one is tangent 
it follows that 
Using the expressions (2) for the 
condition 

numbers Ai are equal. To do this, we use that 
as soon as one of the vectors involved is tangent 
to M» In particular, for all i, j, k. l = 1 ..... n- 1. 

curvature tensor/ of (TII. gs), this lea.ds to the 

0=  (g((2»_L)-.fi)-g((,)-i)-ï»(.)-L)) ,,2 - 
Switching the indices i and j. as well  k and I, we find 
Using the symmetries of the curvature tensor, it then easily follows that Ai 2 = Ai 2 
Summarizing, we have 
Lemma. If Thl  ,I x M2 is a diagonal decomposition et (x, u) u,ith dira 5I = 
•  and dimM2 = n - 1. then there exist orthonormal bases {X,...,X,} and 
is given by 
X1 h + A)] , .... X._  + A)_ t, X,, h 
and an orttogonal basis for T(z,u)M2 is given by 



2890 E. BOECKX 

ReTnark 1. The number ) has a clear geolnetric meaning. Take a nonzero vertical 
rt--1 
vector U at (x, t)" U = i=  and a nonzero vector V tangent to M2 at (x, u): 
= i= fli (AXi h - t). The angle between thc two vectors h cosine given by 
«o() = - E 
By the Cauchv-Schwarz inequality, we bave 
1 < cos(U) < 1 
fi + A: - fi + A: 
with equality if and only if (a,,... ,a-l) and (fil,---,Ç-I) are proportional. 
conc]ude that the angle  t)etwccn lTrI(z,) and T(z.)II2 is such that cos 
1/ + A 2 or tan  = A. So, A determincs the angle between 1 "ThI and Iœee at (x, 
(and hcnce also between I'T[ and hI at that point). 
Hemark 2. Actually, we can give a st.ronger formulation of the lemma. To see this, 
consi(tcr the mapl)ing " T¢.)M  I'TM(z,)" X h + l-t  l-t. Clearly, this 
ma.l)l)ing is linear an(l onto-onc on (Xh) .  restrict  to (X,,h)x  T(x.)SI 
and dcfine the linear mapping 
A" a   X x" Y  An.(x,)(n  ) 
whcre, as 1)cforc, " TrM  M is the natural projection map. Since 
.a =  .(.)(,-')')=  .(.)((.X',  + Y=*)/) = 
the map A is an isometry from u  to X . It associa.tes to a vector X, orthogonal 
to X, the unique vector ', orthogonal to u, such that Xh+ Al t is tangent to 
at (x, u) (or such that AX h - yt is tangent to ,I2 at (x, a)). So, in the lemma, 
we can actual]y choose an arbitrary orthonormal basis {X1,..., X-I} of X  (or, 
alternatively, an arbitrarv orthonormal basis {) ..... )_} of u).  will use 
this possibility in the subsequent subsections. The vectors X (up to sign) and u, 
on the other hand, are determined geometrical]y. 
4.2. Curvature conditions. Since (T,.M, gs) is a (local) Riemannian product, all 
the expressions of the form (U. V)W are zero when U is tangent to hl and W is 
tangent to Iœee at (x, u). Using thc curvature fornmlas (2), this leads to a number 
of curvature conditions for the manifold M. Wc list some of these now. om now 
on, indices i. j, k and l belong to {1,... ,n - 1} unless stated otherwise. 
The tangential and horizont al components of (X h.  ) t) (AXkh _  t) give fise 
to 
2 
(3) n(.X', ÆX')Y) -  g(n(.X, X»), ) = 
while (X h, ,kÇ-h)(AX h -- tt) = 0 leads to 
4 
(5) 
(6) 



WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2891 

Considering/(X/h + At'i t. }'jt)(/xkh -- t) : 0. We obtain 
2 
(7) 
4 
= 
(8) 2(vx,)(..))x+2(),)x+ (..))(.)x 
Finally, from (Xi h + t, Xjh)(Xk h _ t) = 0, we derive 
(9) 
2A  
These conditions can be rewritten in an casier form. To start, we take the inner 

product of (3) with t}. This gives 
2((.\',,, .\-) k;.,  ) 

This is equivalent to 

= -(t(, )t(, i)x,.\). 

(11) 2R(k-, t))X + R(,t))R(u.t))X, =-g(R(a. Yj)X,,R(u,t))X,)X.. 
By interchanging the indices j and l iii this expression and adding both formulas, 
respectively subtracting theln, we get 
(12) R(u,I-)R(u,I))X,, + 
= -2(n(. ? )x. (.  i)x,) x, 
(13) R(u, t)-)R(, l))X. - R(u. t ))R(u, t ))X. = 4R(I ). t-)X. 
Substituting (11) iii (4), we filld the simpler form 
(14) 2A(Vx.R)(u. 
Next, we substitute (3) in (5) to obtain 
Taking the inner product with t, we get 
= 
= 0 



2892 E BOECKX 

by (12). Hence, 
(15) (V.\-kt)(_¥n,.¥j)tt = 0 or. equivalently, 
Sul)st.itutiug (14) aud (15) lu (6), we find 

(16) 

(Vx.R)(. i)xn =0. 

1 
._ ( R(. Y )X,.. R( .. Y )X,,) 
= R(.\'n, Xj)X + 2(,,. (X., 
- (-,,. R(X» X).)X + 
lu order to rewrite (7). we proceed as with (3): we take the inner product with 

aud we use curvature properties to obtain 
4 
(1) 2R(,)X, + :(..Y):(,.'4)X =  (»Xç- X). 
(Note that we also ueed (11) to know that the left-hand side in (17) is orthogonal 
t.o X.) Again switching the iudices j and I and adding and subtracting the two 
fi)rmulas, we get 
4 
(19) R(u,))R(u.}})X,- R(u,}})R(u.))X, = 4R()},))X +  
SM)st.itutiug (]7) and (19) in (8), this rcduces to 

(2o) (v x,R)(, ))x = 
or equivalently, via (15), to 
(21) A(Tx, )(Xk, -\')u = 

• ,( - 1) 
r (&X - X), 

2( - - 1) 
r2 ((ik Yl -- (ilYk). 

It is now easilv verified that (9) is a consequence of the above formulas. As 
to (10), usiug (17) and (20), it simplifies to 
(2) 4EE(x. x)x + R(, (x, x-))x- (. (%., x))x 
In the rest of this section, we wi]] on]v need the Nrmu]as (12), (13), (16), (18), 
(19) and (22). 
4.3. Clifford structures. Putting j = 1 in (12) and (18), we see that 
4 
Since (, ) is a skew-symmetric operator, the nonzero eigenvalues of R(u. 
must bave even multiplicity. Hence, 
*if n is even, the eigenvahm -4/r 2 has even multipli«ity n - 2 on {Xj, X} . 
Hence. the eigenvalue corresponding to X must be zero. This implies that 
n**,Y))X = 0 for  =  .... ,, - . By (3), so n()),)X = 0 for 
j, k = 1 ..... n - 1.  conclude that X,, belongs to the nullitv distribution 



WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2893 

of the curvature tensor R. |n this case, the conditions (12), (13) and (16) 
are trivially satisfied; 
if n is odd, the eigenvalue -4/r 2 has odd multii)licity n- 2 on {X, X,} ±. 
So, the eigenvalue corresl)onding to X, nmst be -4/r 2 as well. Hence, it 
follows that IR(u. }))X,I 2 = 4/r 2 for j = 1 ..... n - I. Bv Remark 2, we 
even have IR(u, Y)X,I 2 = 4/r 2 for every mfit vector 7t" orthogonal to u. Po- 
larizing this identity, we obtain 9(R(u. I')X,, R(u. Z)X,) = (4/r 2) 9(Y, Z) 
for all vectors V a.n(l Z orthogonal to u. In l)articular, the right-hand side 
of (12) equals -(85jt/r2)X,. In this case, conditions (12) and (13) are 
included in (18) and (19) if we allow the index i to I)e n. 

Next, we put i = j -¢ l in (18). Since R(u, 7t))Xj = 0 (this follows ri-oto 
R(u,I'))2Xj = 0), we obtain R(u.}'j)R(u,})Xj = (4/r2)Xt. Avplying the op- 
erator R(u, I)) on both sides, we have 

or, equivalently, 

Since the right-hand side of this expression vanishes both when n is odd and when 
 is even. we conclude 

(23) 

for j, l = 1 ..... n- 1. 
Ve are now ready to discover Clifford representations in our fornmlas, in partic- 
ular in (12) and (18). First, consider the case when n is even. For j = 1,...,  - 1, 
define the operators Ri acting on V" = T,]II by 

T 
R, = _ (, }i) - (x,, .)x, + (x, .)x, 

wllere (-,) = gc. In particular, it follows that R,Xi = \'n, RX = -Xi and 
RXj = (r/2)R(u, ;)X-, j ¢ i. Clearly, R, is a skew-symmetric operator and 
Ri 2 = -id. 



2894 E. BOECKX 

For i ¢ j ¢ k ¢ i, we calculate: 
(Ri o Ri + Ri o Ri)Xn = -RiXj - RjXi 
=-ï-(R(u,Y)Xj+ R(u,}))Xi)=0. (by (23)) 
2 
(Ri c Ri A- Ri o Ri)Xi = Ri(/(u, })Xi) A- 
T 2 
= --f R(u, Yi)R(,Yj)Xi- Xj =0 (by (18)) 
• ,. r R(u, E)Xk) 
+ (,5){(,;-)X-g(R(,})X,Xj)X}) 
T 
4 
=0 (by (s) la (23)). 
So, for i,j = 1,..., - 1. the operators R, satisfy 
R o R + R o R, = -2 iid 

and they correspond to a Clifford representation of an (n - 1)-dimensional Clifford 
algebra on an n-dimensional vector space. 
It is well known (see, e.g., [1] or [3]) that a given real Clifford algebra, say of 
dilnension m, has onlv one (if m - 3 (mod 4)) or two (if m =- 3 (mod 4)) irreducible 
representations and that the dimension n0 of the corresponding irreducible Çlifford 
module is completely determined by m. This relationship is given in the following 
table. 

rn 8p 8p+l 8p+2 8p+3 8p+4 8p+5 8p+6 8p+7 

no 2 4p 2 4p+I 2 4p+2 2 4p+2 2 4p+3 2 4p+3 2 4p+3 2 4p+3 

For a reducible Clifford module, the dimension is a multiple kno of the number n0 
corresponding to the appropriate Çlifford algebra. 
In the present situation, we bave m = n- 1 and kno = t for even . Therefore: 
• if  = 8p: 8p = k24p-  and hence p = 1, k = 1 and n = 8; 
• if n = 8p + 2: 8p + 2 = k24p+l and hence p = 0, k = 1 and n = 2; 
• if  = 8p + 4: 8p + 4 = k2 4p+2 and hence p = 0, k = 1 and n =- 4; 
• if  = 8p + 6: 8p + 6 = k24p+a, which has no solutions. 
Next, suppose that  is odd. Now, we define operators Ri, i = 1 ..... n - 1, 
acting on V n+ = TxII  NXo 

T 
R; =  n(, y)- (Xo, .)x, + (x,,-)x0 



WHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2895 

where ( -, .) = .q. @ go with go(aXo, bXo) = ab. Precisely as before, we show that 
1 i o lj + Rj o 1 i = -25ia id for i, j = 1, .... n - 1. So, we bave again a Clifford 
representation, this time with m = n - 1 aud/,"n0 = n + 1 for odd n. Therefore, by 
the table al)ove: 
• if n = 8p + 1: 8p + 2 = k2 4v altd hence p = 0. k = 2 and n = 1: 
• if,=Sp+3:8p+4=k2 4v+2andhencep=0,k=land=3: 
• if n = 8p + 5: 8p + 6 = k2 4v+a, which has no solutions: 
• if  = 81»+ 7: 8p + 8 =/,'2 4v+a and hence p = Il. k = 1 and  = 7. 
We conclude from this subsection that diagonal decompositious can only occur 
when the base manifold has dimension 2.3, 4, î or 8. (The case n = 1 is irrelevant, 
since then T,.M has dimension equal to one and no decoinpositions exist.) 

4.4. The relnaining dimensions. 

Case n = 2. In this situation, we have a two-dimensional manifold for whiçh the 
mfllity vector space of the curvature tensor is non-trivial. This implies that the 
curvature tensor is identically zero and the space is fiat. 
Conversely, sin«e any tangeut sphere tmndle of a fiat surface M2(0) is a fiat three- 
dimensi(mal space, a diagonal decomposition actually exists around each point (x. u) 
of TM'(O). Note, however, that we also bave T]II2(O) - M")(O) x S(r) with 
{x} x S(r) _ a--l(x). So. TM'(O) also admits a vertical decomposition. 

Case n = 3. Let Xa 1)e the unique unit vector (Ul) to sign) such that Xa h is 
tangent to kl at {x, u). Pick a Ulfit vector X orthogonal to Xa and let  be the 
corresponding unit vector orthogonal to v {i.e., .¥h +A)-t is tangent to .iii). l¥oln 
the comments at the beginlfing of Subsection 4.3. we know that (r/2)R(u.))Xa 
is a unit vector, which is moreover orthogonal to X1 and Xa. So, we obtain an 
orthonormal basis {Xt,X2, Xa} by defining X2 to be X: := (r/2)R(u.'a)Xa. Let 
t) be the corresponding unit vector orthogolml to u and 1". (Since each 1 is fixed 
together with its correspouding Xi, we will hot luentioli this explicitly anvlnore in 
what follows.) 
Using the properties of the operators R( u, ') and (v. ), we then deduce that 

(24) R(u, Y1 )x = 0, R(u, Y )x = - -. x3, 
/(, ?)x, = 2_ xa, R(u, ?)x2 = o, 

and from (13) and (19) it follows that 

R(tt "1.)X 3 - 2_ 

Next, we COlnpute R(X, Xj)X, i, j. k = 1.2.3, ffonl the equalities (16) and (22). 
writing R(u,R(Xi, X)u)Xa as Y'9(R(u, Yz)Xi, Xj) R(u,})Xt. and using (24)and 
(25). This gives 

(26) 

1.2 R( X1, 
1. R(X,, X3) 
1.2 R(X2, X3) 

Xa X2 Xa 

-AX AX 0 
-CX3 0 CX 
0 -CXa CXa 

where A = (,4 _ ,2 q_ 1)/),2 and C = (aA 2 + 1)/ 2. 

(25) t(Y,)xl=-x2, (,)x.=xt, (Y,)xa=o. 



2896 E. BOECKX 

Since both {X1, X2, X3} and {¥], t), u/r} are orthonornial bases for Txll, there 
is an orthogona] matrix Q = (qij) 6 0(3) such that 

X., = Q Y., • 

Changing Xa to -Xa if necessary, we ma,y even sut»pose that Q 6 SO(3). Then 

R(Xt, A'3) 

= (qlqa2 -- qt2q31) H(}], }')) + qllq33 --ql3q31 
?- 
+ q12q33- q13q32 R(}), 
= -q3 R(Y1, t;_) - q2___ R(u, ti) + q__l R(u, 
T F 

If we let both sides act. on X1, ¥._, and X3 and if we use (24), (25) and (26), we 
find that 

q21 = -C/2. q2 =0, q3 = 0. 

Since Q  S0(3), it follows that qe2 +qe22 ÷q23 2 = 1 and hence 1 = (3A2+ 1)/2A 2 
or A - + 1 = O. which is a contradiction. Hence, no three-dimensional manifold 
adnlits a diagonal decomposition of its tangent sphere bundles at anv point. 

Case 11 = 4. Let X4 be the unique unit vector (up to sign) in the nullitv distribu- 
tion of R,. Take two mutuallv orthogonal unit vectors X and X.,,_ perpendicular 
to X4. Since (r/2)R(u, Y1)-\'2 is a unit vector and orthogonal to X1, X2 and X4, we 
can define X3 := (r/2)R(u, t])X2. From the properties of file operators R(u. 
i = 1,2, 3, it follows that 

(27) 

r R(,,, "1) 
," R(u, t&) 
,'R(u, t5) 

X1 X2 Xa X4 

0 2Xa -2X.2 0 
-2X3 0 2X 0 
2X -2X 0 0 

Next, we decompose Xa with respect to the basis {tq, t.), t, u/r}: 

X4 = qt] + q2I') + q3} + qa -, 

q + q2  + q32 + q42 = 1. 

Then R(u, Xa) = ql R(, }'1) ÷ q R(u, Y.,_) + q3 R(zt. }5)- Since Xa belongs to the 
mfllity distribution of R, this operator vanishes identically. By (27), we nmst bave 
q = q2 = q3 = 0. Hence, Xa = :t:u/r. But this is impossible, since u clea.rly does 
hot belong to the nullity distribution. So, also for four-dimensional mallifolds, a 
diagonal deconlposition of its tangent sphere bundles does hot exist at anv point. 

Case n = 7. The argtlnlent for n = 7 goes along the Saille lines as that for n = 3. 
but it is more involved technicallv. Again we start with the trait vector Xr, uniquely 



VHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2897 

deternfined up to sign. su«h that Xr h is ta.lgent to 2I, and with an arbitrary unit 
vector X, orthogonal to X7. The unit vector X2 := (r/2)R(u, }',)X7 is orthogonal 
to both X aud X7. Then it follows t.hat 

R(, ;)x, = o, 
R(u, }))X = - Xr, 
,ï, 

R(u. }')X2 - 2 X7, 
R(.u. )x2 = o, 

R(-u, Y)X7 = - X2, 
R(u, ')Xr -  " 
-- -TX. 

Note that R(u. Y1) aud R(u, t?) preserve spal,{Xl,-\'2, X7}, hence by skew-sym- 
metry also its orthogonal complemeut. Next, take a unit vector X4 orthogonal 
to X1, X.2, Xr and define the unit vectors Xs := (r/2)R(u,t))XŒE aud X := 
(r/2)R(u, t'I)X4. Then Xs and X are ah'eadv orthogoual to X, X2, X4 and Xr. 
Further, 

£2 
i-2 
-- g(R(u,}])R(u,}'2)X4, X4) 
4 
£2 
(28) = g(R(u.I))R(u,}i)X4,X4) (by (18)) 
r 2 
-- g(R(u, })X4, R(,u, )X4) 
4 
= -{Xs, 

and X5 and X6 are mutually orthogonal as well. Finally, sin«e R(a, ]q)X5 is or- 
thogoual to X, X2, Xs, Xr and 

we mav defil,e X3 := (r/2)R(u, I'I)X. 
In this way, we have defined ml orthonormal basis {X1 ..... Xr}, and the actionb 
of the operators R(u, }), i = 1 ..... 6, can be COUlputed explicitly in this basis using 
the properties (12), (18) and (2a) above. We obtain 

• r R(,, \) 
• r R(u, 
" R(-u, 
• r R(u. 
r R(-. 

X X.2 X3 X4 X5 X6 Xr 

0 -2Xr -2X5 2X6 2X3 -2X4 2X2 
2Xï 0 2¥6 2X5 -2X4 -2X3 -2X1 
2X5 -2X6 0 -2X7 -2Xt 23£2 2X4 
-2X6 -2X5 2X7 0 2X2 221 -2X3 
-- 2X3 2X4 2X1 - 2X2 0 - 2X7 2X6 
2X4 2Xa -2X2 -2X1 2Xr 0 -2X5 



2898 E. BOECKX 

Next, ve calculate R(}]-, }-)Xk from (13) and (19): 

X1 X2 23 X4 X5 26 X7 

--222 2X 2X4 -2Xa 226 -2X5 0 
-2Xa -224 22 222 0 2XŒEE 
-2X4 223 -222 2X1 227 0 -2X5 
-225 -2X6 0 -227 221 222 224 
-226 2X5 -2XŒEE 0 -222 221 2Xa 
224 -2Xa 222 -22 227 0 -225 
-223 -2-\'4 221 222 0 -227 2\'6 
2X6 -225 -2X7 0 22 -22 223 
-225 -226 0 227 22 222 -2Xa 
2X2 -22 -224 223 226 -225 0 
0 227 -2-¥5 -226 223 224 -222 
227 0 -226 2X5 -224 223 -2XI 
227 0 226 -225 224 -223 -2XI 
0 -227 -225 -226 2X3 224 
222 -2Xt 224 -223 -226 225 0 

Using (16) and (22), ve ca,, now compute the «,r'«ature components R(Xi, 2j)Xk 
for i,j,k = 1 ..... 7: 

r 2 R(X 
r 2 R(X1 
r 2 R(X 
r 2 R(X1 
r 2 R(X1 
r 2 R(X1 
r 2 R(X2, 
r 2 R(X2, 
r 2 R(X2, 
r 2 R(X2, 
r 2 R(X2, 
r 2 R(X3, 
r 2 R(X3, 
r 2 R(X3, 
r 2 R(X4, 
r 2 R(X4, 
,,.2 R(X4, 
r 2 R(Xs, 
r 2R(X6, 

X1 22 X3 X4 X5 X6 X7 

-A22 AX 224 -223 226 -225 0 
- B23 24 BX - 22 0 0 0 
-B24 -X 3 22 BX 1 0 0 0 
-BX5 X6 0 0 BX -X2 0 
-BX6 -X5 0 0 22 BX 0 
-CX7 0 0 0 0 0 CX 
-X4 -BX3 BX2 X 0 0 0 
23 -BX4 -Xt BX2 0 0 0 
-X6 -BX5 0 0 BX.2 X 0 
25 -BX6 0 0 -X BX2 0 
0 -ex 7 0 0 0 0 C. 2 
222 --221 -AN4 A23 226 -2X5 0 
0 0 - BX5 26 BX3 -X4 0 
0 0 -BX6 -25 24 BX3 0 
0 0 -CX 7 0 0 0 CX 3 
0 0 -X6 -BÆ5 B24 X 3 0 
0 0 25 -BX6 -23 BX4 0 
0 0 0 -CX7 0 0 CX_ 
222 -2X 2X4 -223 -AX6 AX5 0 
0 0 0 0 -CX7 0 CX5 
0 0 0 0 0 -CX7 CX6 

where A = (A 4 - A 2 + 1)A 2, B = (A 2 + 1)2/A 2 and C = (3A 2 A- 1)/ 2. 
We now show that the tables above are incompatible. To see this, we relate 
the two orthonormal bases {X ..... 27} and {} ..... Y6, u/r} by an orthogonal 



WHEN ARE TANGENT SPHERE BUNDLES REDUC1BLE? 2899 

transformation. Let Q = (qij)  0(7) be such that 
Putting Q, := qi.qj - qqj, we thon have the equality 
6 6 
ij 
k<l=l k=l 

No, 

and 

2 

= r g(R(X,X2)X*,X4) 
6 6 
k</=l k=l 

= (QI - Qàï + Q) 

u)X3, X) 

12 
2 : .2 g(I(XI,X2)X5,X6) _ 2(Q22 ÷ Q3142 _ Q56)- 
This implies that Q] = 1. Now, using the Cauchy-Schwarz inequality and the fact 
that Q is orthogonal, we find that 
1 = QI = qllq2-. - 
Hence, ail the inequalities must be equalities. In particular, we have q13 .... 
ql7 = q23 .... = q27 = 0 and consequently 
X =cosO l +sinO )), X.2 =q(-sinO1Yl +cosO I) 
where ci = :t:1 and 01 is some real number. In a similar way, we can show that 

Q = Qs566 = 1 and that 
X 3 --- COS 0 2 }] ÷ sin 02 
X5 = cos03 t' + sin03 Y6, 

X4 = e2(-sin02 l + cos02 
X6 = 3(- sin 03 } ÷ cos03 Y6)- 

As a consequence, we also have X7 = cuir, e = -1-1. Using the tables above, we 

find that 
0 

= r e R(X, XT)X3 = -e (cos0 r R(u, }'1)X3 ÷ sin01 r R(u, 
= 2e(COS01Xs - sin01 X6), 
which gives a contradiction. So, also seven-dimensional manifolds cannot bave a 
diagonal decomposition for their tangent sphere bundles at anv point. 

Case n = 8. This case is treated as the case n = 4, but the appropriate choice for 
the basis {X1,..., Xs} requires a little more tare. Let Xs be the unique unit vector 
(up to sign) in the nullity distribution of R and take two arbitrary unit vectors 
X and X2 that are mutually orthogonal and perpendicular to Xs. As before, we 



2900 E. BOECKX 

define X3 := (r/2)R0*, })X2, which is a unit vector orthogolml to X1, X2 and 
It follows that R(u. }))Xs = 0 for i = 1.2, 3, and 

n(.. }i)xl =0. 
R(U. l.))X1 -- 2 3, 
R(u, Ya)X1 = 2_ 

Because they are skew-symmetric, the operators R(u, }i), H(u,}:z) and R(u.};) 
also preserve l I = { X 1, X2, Xa, Xs } ±, and on t his space t hey ant i-conmmt e by (18). 
It is easv to check that the operator (ra/8)R(u, I])R(u, I))R(u. }5) is symmetric 
on II" and that it squares to the identity on Il . Hence, it bas a basis of eigenvectors 
corresponding to the eigenvalues +1 and -1. Let Xa be a unit vector in Il" such 
that r a R(u, Yt)R(u, Yz)-/ï'(u, }5)Xa = 8eX4 where e = +1, and define 

r/(. };)X. 

Clearly, X5, X0 and X7 are unit vectors orthogonal to X1, X2, X3. X4 and Xs. 
A comlmtation similar to (2) shows that they are also orthogonal to one another. 
So, we have an orthonornml t)asis {X1,...,Xs} for T3I. It is now possible to 
compute explicitly the action of R(u. ), i = 1 ..... 7. from the condition (18). VVe 
get 

• /(, i) 
r R(u, Y2) 
-/(u.}) 
.,-/(.. ;) 

X X2 X3 X4 X5 X6 X7 Xs 

0 2-\'3 -2X2 2X5 -2Xa -2eX7 2e-\'6 0 
-2X3 0 2X 2N6 2eX7 -2Xa -2eX5 0 
2X2 -2X1 0 2X7 -2eX6 2eX5 -2X« 0 
--2X5 -2X6 -2X7 0 2XI 2X2 2X3 0 
2X4 --2¢-\'7 2cA'6 -2.Y1 0 -2e-¥3 2e-\'2 0 
2eXz 2Xa -2eX5 -2X2 2eXa 0 --2eX1 0 
-2eX6 2eX5 2X4 -2Xa -2eX 2eX1 0 0 

Next, decolnpose X8 with respect to the basis {I],..., I), u/r}: 

Xs=qll]+---+qŒE})+qs-. ql 2+.--+qs  = 1. 

Since X8 belongs to the nnllity distribution of the curvature, we have 

7 
o= t(,x) = , q t(.) 

and ffom the table above we deduce ql = .... q7 = 0. Hence. XS = Tu/r, but 
this is impossible since u does hot belong to the nullity distribution. So, also in 
the eight-dimensional case, a diagonal decolnposition of the tangent sphere bundles 
does not exist at even a single point. 

Remark 3. The operator r 3/(u. ]/)]:(u,})/(u. Y3) acts as 8eid on the vector 
space spanned by X4, X5, X6and Xr, as is seen easilv froln the previous table. 
The two different cases, e = +1 and e = -1, correspond to the two non-equivalent 
irreducible Clifford representations of the seven-dimensional Clifford algebra. 



\VHEN ARE TANGENT SPHERE BUNDLES REDUCIBLE? 2901 

5. VERTICAL DECOMPOSITION 

Now, we sui)pose that we have a vertical decolnposition T,.III - III1 × 3I such 
that I'T,.I(,) C T(,)I cverywhere. In this situation, if (x. u)  ,ll  {q} 
for SOlne q  M2, then -(x) C I x {q}. Çonsequently, we bave AI l X {q} = 
--1(7(/l  {q})). SO, th lçve [1  {q}, ('orrspolldllg to th prodllçt, project 
under  to a foliation L on (M.g) and n-(L) = {Mt x {q},q  Al2}. Let 
L be the distl'i]mtioli Oli Al tangent to L. I)cfinc the distribntion L2 to bc the 
orthogonal distrit)ution to Lt on I. Then 
where h denotes lhe horizontal lift. 
If X and ]" are vector fields on I tangent to L and if, i are tangent to L2, 
then X h, yh are tangent to 1 and rh, V h are tangcnt to I2. Because of the 
product strncture, we have that çx - and çg X  arc t311gcllt to [1 lld Çg i h 
and Ç.x' Un are tangent to AI2. Using the expressions (1) for Ç, this means that 
• Ç.x]" and ÇuX are sections of L" so, L is totally geodcsic and even 
totally parallcl: 
• ÇUl" raid ÇA-U arc sections of L2: so, also L2 is totally geodesic and totally 
parallel (iii part icular, L is integrable with sociated foliation L2); 
• R(U, V)u = R(X, U)u = 0: so, L2 is containcd il the nullitv distribution 
of the curvature. The leaves of L2 are therefore fiat. 
These properties imply that L and L.2 consist of the leaves of a local ielnamiian 
prodnct M  M' x N  where k = diln L  n (sec [8]). 
Suppose converselv that I  is locally isometric to M' x N  for 1  " 5 n. This 
gives rise to two foliations on I: L = {M'x {v}. t, G N  } and L2 = {{p} xN,p G 
I}. Define two COlnplelnentary distributions L and 2 on TI by 
L = l'Trl  h(TM'). L = h(TNk). 
It is easily checked sing (1) that  and {2 are totally geodesic and totally parallel 
complementary distributions. Hcnce, the leaves of their correspollding foliations È 
and  are actually the leaves of a local Riemanniall product, hl particular, note 
that  = {-(3I' x {t'}),v  Na}. So. TM is indeed locally reducible. 

6. (LOBAL RESULTS 

YVe continue with the notation of the previous seçtioll. In order to derive results 
concerlling the global reducibility of (TriiI, gs ), we will exploit the relationship 
between the foliations L1 and L2 of (il1, g) and the foliations t and 2 of (Tr]II. gs) 
iii t.he case of a vertical deconposition. "lVe ha.ve alreadv relnarked that L1 and 1 
deternfine each or her reciprocally by L 1 = re (! 1 ) and   = re- l L 1- The relat ionship 
between the foliations L2 and _ is hot so straightforward. We still have L = 
rc(Ï2), but deternfining L2 from L requires a little more tare. To construct the 
leaf ç of 2 through a point (x, v)  T,.III, consider all the curves iii the leaf S 
of L_ starting at z  III. Then, ç consists of all end-points of the horizontal lifts 
of these curves starting at (x, u). We call ç the horizontal lift, of S through (x, u). 
Since  is evervwhere horizontal, the lnap re-  -- S is a local isolnetrv and , is 
a Riemalmian covering of S. \¥hen S is silnply COlmected. 5} and S are globally 
isolnetric and. iii particular, one-to-one. 



2902 E. BOECKX 

\Vith these comments in nfind, we now proceed to the proof of the Global The- 
orem. So, we suppose that dira M  3 and that (T,.M. gs) is isometric to a global 
Riemalmian product (kl, g) x {M2, g2). Since dira M  3, this is a vertical decon> 
position and I'T,.M is tangent to one of the fmtors, say M. Cnsider kl and 
as submanifolds of TAI {i.e., choose one leaf of eh of the foliations L and 
and define the submanifolds M' := (kl) and F := (M2) of M. From the local 
considerations in the previous section, we know that (M.g) is locally isometric to 
the Riemannian product M' x F and that F is fiat. 
We show that there is a onc-to-one correspondence between M and M' x F. Take 
a. point x  kl and consider an arbitrary vector tt E Tkl of length r. Through 
(.. u) e Tal, there is a unique leaf = of L and a unique leaf 2 of L2- Because 
of the product structure on T,.M,  cuts Al2 in a unique point O  A& and 
 cuts al, in a torique point   al,. Put p := ()  ai' and q= := (0)  
 claire that lhe con'espondcnce kl  al' x F" ,,"  (v=, q) is well-defined, i.e., 
independent of the choice of the ta.ngent vector u. To ste this, take another vector 
,,  Tal ,f length r. Sin«e (, u) = a" = (.r. t,), the leaf  of 1 contains both 
(.r, t,) and (,r, v); so we have u = ,'1, 0, = 0,. and q,, = q,,. The unique leaf 
of 2 through (a', t') is different from ç2.- However, both are horizontal lifts of 
S2, = w(2u). So, if #(t) = (.r(t),u(t)) is a curve in ç2u such that (0) = 
and (1) =,,  ,/I, then 7 = ° runs flom,r tot  M in $2,. Denote 
by  the horizontal lift of 7 starting al (a',v). Çlearly,  lies in 2 and ends 
On the other hand. starting from a couple (p. q)  M' x F, we find the corre- 
sponding point ,r  al as • = u(, 0) for some   al with () = p and the 
unique 0  M.e with (0) = q- Via an argmnent as above, one shows that • does 
hot depend on the choice of  and that the nmp (p, q)  •defined in this way is 
the inverse of the map •  (p, q). 
Next, we note that the correspondence kl  M' x F: •  (p,q) is defined 
so as to respect the local product structure. In particular, the metric  on 
corresponds to the product metric of M' x F, and the first statement is proved. 
Conversely, suppose that (M, ) is the global product space (M', ') x (F, g0)- 
By choosing a leaf of both product foliations, one can consider M' and F  sub- 
manifolds of M. Let «0 be their intersection point and choose a vector uo  ToM 
of length r. Define kl as the inverse ilnage of M' under the projection  and 
k12  the horizontal lift of F through (a'0, u0). Since we suppose F to be simply 
connected, kl. is isometric to the fiat space (F. 0) and ,1 and I2 have {0. 
 unique intersection point. 
We show that there is a one-to-one correspondence between TkI and kI x 
Take {, u)  TM and denote by S1 the unique leaf of L 1 Oll al through m and by 
$2 the unique leaf of L2 on kl through  Then, the leaf  of  through ( 
is given by -(S) and the leaf 2 of 2 through (, u) is the horizontal lift of 
through this point.  cuts a& in a unique point O with (0) = S  F, and 
2 cuts kl in a unique point  with () = $2  al'. (Note that the simply 
connectedness of F is essential to ensure uniqueness.) Çlearly, the correspondence 
Tkl  1 x M2" (, u)  (, 0) is well-defined and it is hot dicult to construct 
it.s inverse. Since this correspondence also respects the local product structure, the 
metric gs on TM corresponds to the product metric on kI x k&. This completes 
the proof of the Global Theorem. 



WHEN ARE TANGENT SPHERE BUNDLES FIEDUCIBLE? 2903 

Remark 4. The proof of the Global Theorem continues to hold when n = 2 for the 
case of a vertical global decomposition of (TAI, gs). Clearly, the base manifold is 
then fiat.. That we need the simply commctedness of the fiat factor tan be seen 
from the example of a two-dimensional fiat cone C The vertical and horizontal 
distributions on TrC are both integrable, and locally their integral manifolds are 
the leaves of the local product foliation on T,.C If it were a global decomposition, 
everv maxinml integral manifokt of the horizontal distribution would intersect ev- 
ery vertical fiber exactl.v once and it would be isometric to C under the natural 
projection 7r. This would define a global parallelization of C, contrary to the fact 
that its full holonomy group is non-trivial. 

IEFERENCES 

[1] M. F. Atiyah. R. t3ott and A. Shapiro, Clifford modules, Topology 3, Suppl. 1 (1964), 3-38. 
MR 29:5250 
[2] J. Berndt, E. Boeckx, P. Nagy and L. Vanhecke, Geodesics on the unit tangent bundle, 
preprint, 2001. 
[3] J. 13erndt, F. Tricerri and L. Vanhecke, Generalized Heisenber 9 groups and Damek-Ricci 
harmonic spaces, Lecture Notes in Math. 1598, Springer-Verlag, Berlin, Heideiberg, New 
York, 1995. MR 97a:53068 
[4] E. Boeckx and CI. Calvaruso, When is the unit tangent sphere bundle semi-symmetc?, 
preprint, 2002. 
[5] E. Boeckx and L. Vanhecke, Characteristic reftections on unit tangent sphere bundles, Hous- 
ton J. Math. 23 (1997), 427-448. MR 2000e:53052 
[6] E. Boeckx and L. Vanhecke, Curvature homogeneous unit tangent sphere bundles, Publ. 
Math. Debrecen 53 (1998), 389-413. MR 2000d:53080 
[7] E. 13oeckx and L. Vanhecke, Harmonic and minimal vector fields on tangent and unit tangent 
bundles, Differential Cieom. Appl. 13 (2000), 77-93. MR 2001f:53138 
[8] CI. de Rham, Sur la reductibilité d'un espace de Riemann, Comment. Math. Helv. 26 (1952). 
328-344. MR 14:584a 
[9] O. Kowalski and M. Sekizawa, On tangent sphere bundles wth small or large constant radius, 
Ann. Cilobal Anal. Geom. 18 (2000), 207-219. MR 2001i:53049 
[10] O. Kowalski, M. Sekizawa and Z. Vl£ek, Can tangent sphere bundles over Riemannmn 
mani]olds bave strictly positive sectional curvature?, in: Cilobal Differential Cieometry: The 
Mathematical Legacy of Alfred Ciray (eds. M. Ferngndez, J. A. V'olf), Contemp. Math. 288, 
Amer. Math. Soc., Providence, RI, 2001. 110-118. MR 2002i:5304ï 

DEPARTMENT OF h[ATHEMATICS, KATHOLIEKE UNIVERSITEIT LEU\ EN. CELESTIJNENLAAN 200B. 
3001 LEUVEN. BELGIUM 
E-mail address: cric. boeckx@wis, kuleuven, ac. ho 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2905-2923 
S 0002-9947(03)03274-4 
Article electronically published on March 17, 2003 

CRITERIA FOR LARGE DEVIATIONS 

HENRI COMMAN 

ABSTRACT. V'e give the general variational form of 
lira sup(./i, e h( x ) / t tta ( dx ) ) t 
foi" any bounded above Borel measurable function h on a topological space X, 
where (tt) is a net of Borel probability measures on X, and (ta) a net in ]0. oe[ 
converging to 0. When X is normal, we obtain a criterion in order to have a 
limit in the above expression fol" ail h continuous bounded, and deduce new 
criteria of a large deviation principle with hOt necessarily tight rate functiom 
this allows us to remove t he tightness hypothesis in various classical theorems. 

1. INTRODUCTION 

Let Cb(X) be the set of real-valued bounded continuous functions on a topological 
space X, (Va) a uet of Borel probability measures O11 .', and (/a) a net in ]0. 
converging to 0. For each [-o% +ec]-vahled Borel measurable function h on X. 
we write ptâ (eh/t) for (fx eh(x)/t° ta(d'r))t«" and define A(h) = log lira pta" (e h/t) 
provided the limit exists. 
The ailn of this paper is to clarify the relation between the existence of A(h) for 
all h  Cb(X). and the one of a large deviation principle for (p) with powers (t). 
This problenl origillates fronl Varadhan's theorenl, which sta.tes that if X is regular. 
then such a prillciple with tigllt rate function .1 inlplies the existence of A(h) for all 
[-oc, +oc[-valued contilmous fimctions h on X satisfying some rail condition (in 
particular for ail h  rb(X)), with moreover A(b) = sup:ex{h(x ) - J(:r)}. This 
theorem is a crucial argument in the proofs of all related results: in particular, these 
also hold under some tightness hypothesis (of the rate fimctiom or of the net (it2 
i.e., exponential tightuess). 
We present here a new approach based on a variational representation of the 
fuuctional lim suppt2(e'/t), and a criteriou of existence of A(-) on Cb(X) wheu 
X is uormal. This leads to functional as well as set-theoretic large deviation cri- 
teria, which allow us to relnOVe the tightness condition in various basic results of 
the theory; moreover, the proofs are nonstandard since there are no compactness 
arguments in the entire paper. Notice that all the results work for general nets of 
measures and powers, and (except for Section 4 and "(i)  (il)'" in Theoreln 3.3 
where X is assumed to be normal) for a general topological states space. 

Received bv the editors January 3, 2002 and, iii revised form, November 9, 2002. 
2000 Mathematics Subject Classification. Primary 60F10. 
This work was supported in part by FONDE('YT Grant 3010005. 

@2003 American Matheiuatical Society 



2906 HENRI COMMAN 

We begin in Section 2 bv stating somewhat unusual equivalent definitions of a 
large deviation principle (Proposition 2.3) which imply the existence of a minimal 
rate flmction. 
In Section 3. we prove a general variational form of lim supp « (e h/t« 1y) for any 
Bord measurable flmction h on X and any Borel set }" C X sa.tisving the (local- 
ized) tail condition of Varadhan's theorem (Theorem 3.1). Bv the saine methods. 
we obtain a sucient (and necessary if X is normal) condition for the existence of 
A(h) for ail h  Ch(X), in the spirit of the Port manteau theorem (Theorem a.a). 
A generalized version of Varadhan's theorem without anv tightness assumption is 
a direct consequence (Corollary 3.4). 
In Section 4, assmning that X is nornml, we look for necessary and sufficient con- 
ditions that A(-) (as a flmctional on Ch(X)) nmst satisfv in order to bave a large de- 
viation principle. This is obtained in Theorem 4.1. which gives rive such conditions: 
in particular, it states that large deviations occur if and only if lin A(hi) = A(h) for 
each increasing net (h) in C(X] converging pointwise to h  C(X). As corollaries, 
va.rions basiç results hold verhatim without tightness ssumptions: this is the ce 
for the eqnivalence imtween the Laplace and large deviation principles (Crollary 
4.2), and for thc variational form of a rate flmction (4.5): a large deviation princi- 
ple is characterized  a convergence in a mrrow space of set-fllnctions much larger 
than capa.cities (Remark 4.5]: Crollary 4.3 improves Brvc's theorem by weakening 
the exponential tightness lgpothesis; (4.2) gives the infimum of a rate flnction 
J on anv closed set i,, terres of lira inf p" (-) (resp. lira sup p" (-)), generalizing a 
well-known expression of J. 

"2. PREL1MINARIES 
Let .T" (resp. Ç) denotes the set of closed (resp. open) subsets of X. For each 
O 
[0. +]-valued function f on X we denote by ] (resp. f) the least upper semi- 
continuous function on X greater than f (resp. the greatest lower semi-continuous 
function on X less tlmn f), and define a map 7I " 9 -- [0.+oo] by 7I(G) = 
sup.eG f(x) for ail G G {7. 
We collect here some characterizations of positive bounded upper senfi-contim- 
ous flmctions that we will use in the sequel. 
Lemma 2.1. Tbere is a bijection between the set of positive bounded upper semi- 
«ovtinuous fvnctios f on X and the set of maps ï " Ç - [0. +OE[ satisfying 
(9.1) 7(OGi ) =- sup',/(Ci) for all {Gi:i  I} C g, 
given by the maps 
• 7  tir(x) = infeç,=e 7(G) for all x  X. 
• f-*7EEf. 
Mor«ov«r, for ch positive bound«d fun«tion f on X. the followin properties hold: 
(i) f = fæ; 
(ii) f is the unique positive upper semi-continuous function h on \" satisfying 
(iii) sup: c f(x) = infG»:Gç Tf(G) for all  C X: 



CIITEIIA FOl LAIGE DEVIATIONS 2907 

Proof. Let f be a positive bounded upper semi-continuous flmction on X. For each 
1'" C X we ha.ve supr f _< Sllp{jï<sup. f+e} f  Sllpt f ÷ 6; since {f < supr f + e} 
is open and contains }', we obtain 

(2.2) sup f = inf 
 GD;Gç 

whence f's = f" \Ve now prove that %% = "y for ail 7" ç --' [0, +oo[ satisfying (2.1); 
let 7 be such a map. , > 0. and dcfine G), = U {G E 
and for all x E G, we have f.c(x) <_ "y(G) <_ u so that [,J<a G,, C {f. < ,}. For ail 
aï  {f. < ,} there is G 9 .c such that "y(G) < ,, and thus 
This shows that {f. < A} C QJ,«a G, and thus {tir < 3,} = QJ«a G, which is an 
open set, so that f. is upper semi-contilmous. Let G 
(with the convention supo = 0). If supa f < 3(G), then G C {f < 
for some e > 0. Since ri is upper Selni-continuons, {f < 7(G) -e} is open 
with {f < 3(G) -e} C G(a)_, and since 7 is clearly ilwreasing, we obtain 
() N 7({f < ()-«})  (G(a)-) N ()-«, which gives the contradictiou. 
Thus 7(G) = sup G f7 for all G  Ç and the first assertion is proved. 
Let f be a positive bomided functioli on .¥. Then 7I is bomMed and satisfies 
(2.1), so that f, is upper semi-contimloUS with fT,  f. For ail positive bomded 
upper semi-contimous fllnctions f  f, we have I 
f, which implies f = f. This prove (i). If h is a positive upper Senli-Colitinuous 
flmction on X satis[ving % = 7I, then h is bounded and (il) follows ri'oto (i). Let 
Y C X. By (2.2) we have sup. f = infGDY.Geç suPG f. and since  = I by (ii). 
(iii) holds. Let h = V{9  [0. +[x: % = YI}- It is easv to see that % = f, and 
since fh is upper semi-contimtous with fh  h. we have Yh = fh = f whence 
h: fTh = f by (i). Thus (iv) holds. 

Definition 2.2. We say that (p) sa.tisfies a large deviation principle with powers 
(t) if there is a lower semi-continuous flmction .1 : X -- [0. :c] such that 
lira sup  (F) _< sup e -d (x) <_ sup e- a (x) _< lira inf l,t« (G) 
xF xG 
for all F  9 r, G  {ï with F C G. Them J is called a rate function for (#t«), which 
is said to be tight if it has colnpact level sets. 
Notice that in the literature, a large deviation principle is in general defined for 
 1/n 
nets (lt)e>0 or sequences t#, ,eN*- In the sequel, when we will refer to known 
results that will be proved again, we will hot make this distinction and state them 
with general nets of measures and powers. 
By (2.3), the following proposition shows that the set. of rate functions for 
bas a minimal element: it is the only Olm if X is regular since it is well known ([2], 
Lemma 4.1.4). 
Proposition 2.3. The following statements are equivalent: 
(i) (#,) satisfies a large d«viation princwle with pow«rs (t,). 
(ii) There is a map 7 : ç--' [0, 1] such that 
(a) limsup#t°(F) N /(G) N liminfpto(G) for all F  .. G  çwth 
FcG. 
(b) "y(Uiel ai) = supiel "/(ai) for all {ai : i  I} C ç 



2q08 HENRI COMMAN 

(iii) There s et fiction f : .V -- [0, 1] such that 
limsupptâ(F) _< sup f(:r) _< liminfltâ (G) 
for «dl F G.. G E Ç u,ith F c G. 
(iv) There is a. fu.7,ction f: X  [0.1] stch that 
limsul,/,tâ (F) _< SUl, f(.r) _< lira inf p°(G) 

Jbr a, ll F G., G G ç with F c G. 
I.f (i) hot&. then (i) holds wth r«te fmctio .1 given 
(2.3) e-J= V{ f G [0.1] "x "f s«tisfi«s (iii) (resp. (iv))}. 
altd 
(2.4) 7¢-a = V{G [0.1] ç "7 satisfies (ii)}. 
q (il) holds u,ith , thon (i) hold.s" with rate f«nction - log G where 
J'v(a') = inf 7(G) for ail z  A. 
(f (iii) (rep. (iv)) bol& with f. thett (i) bol& u,ith rate fu,ctio -logf. 
Proof. If (il) holds with 7- then f. is upper semi-continuous and (i) holds with rate 
fim«tion - log f. by Lemma 2.1. 
If (iii) holds with f, then (il) holds with 7, and so (i) holds with rate function 
- log , since fw = f by Lemma 2.1. 
If (iv) holds with f, then pu 7(G) = suprcG supr f for ail G G ç. and notice 
that 7 satisfies (il). Thus (i) holds with rate fimction -logfT. Since f 
we hRvo 
sup f  sup f  sup f = sup f 
F F G G G 
for ail F  U, G G Ç with F C G (the equality follows from Lemma 2.1 (iii)). 
Thus (i) holds with ra.te fimcion - log f. 
If (i) holds, then (il), (iii) and (it,) hold. The fimçtion 
t, = {f  [0.1] x -f satisfies (iii) (resp. (it,))} 
obviouslv satisfies (iii) (resp. (it,)); the saine for h bv the preceding discussion, and 
h =  by the definition of h; put e -J =  and obtain (2.3). The map 
[0, 1] 9 • 7 satisfies (ii)} satisfies (il), and so (i) holds with rate funct.ion J given bv 
c-J = fTt. Since 7-a = 7t = 7t, (2.4) holds. 
Corollary 2.4. (Contractio principle) Let Y be 
t a coti.uous fu.ctio.. If (p) satisfies a large deviation priciple with powers 
(G) ad rate fueto, jx. the. ([p]) satisfies a large deviatio, pri.ciple with 
0 
powers (G) and rate functio, jv = 1 where l(v) 
Proof. Let jx be a rate fimction for (it ). The relations 
i.n .p [,d  (c) = in p G  (- I(F)) 
_j _d x 
 sup e  sup «  liminf[p]t(G) 
-(F) -(G) 



CR1TERIA FOR LARGE DEVIATIONS 2909 

for all F E .', G E çwith F C G and Proposition 2.3 show that (Tr[tt,]) satisfies 
a large deviation principle with powers (t) and rate functiol JY = -log] where 
jx jx 
f(/) = Sllprr-,(y ) e- for all g G Y (Sillce Sll])r-l(F ) e- = SllPF f). Equivalently, 
jr = l where l(g) = illfr-l(y ) .J\" for ail g G I'. [] 

3. A GENERAL VARIATIONAL FORMULA 
Up t,o now, the onlv known condition that ensures the existence of A(h) for all 
h  Ch(X) (and lnore generally for all [-c, +oe[-valued COllt.inllolls fllnctions ]2 on 
X satisfving the tail condition 
(3.1) lira limstlp/tW (e h/t l{eh>^t} ) = 0) 
is the existence of a large deviation principle with tight rate fimction, and A(h) is 
expressed in a variational form in terres of this rate fimction (Varadhan's theoreln, 
[2] Theorem 4.3.1). In this section, we generalize these results in two directions. 
First, Theoreln 3.1 gives the general variational form of lilnsllPltâ ' (e h/t° lr) for 
any Bord set }" C X and anv Borel measurable flln('tion ]/ on ." satisf.ving the 
localized tail condition (3.2). Next, Theorem a.a gives a sufficient condition for the 
existence of A(-) on Ch(X) which is also necessarv when X is nornlal; moreover, 
the variational form of A(-) is obtained in terres of any set-function 7 6 [0.1] 
satisfying the typical in-between inequalities of large deviations (a.ua). As a COl- 
sequence, Varadhan's theorem is gelmralized in various ways (Corollary 3.2 and 
Corollary 3.4). 
For each nlap h • X + [-oo, q-oo] we put F h - {e h E [ - 6,  q- 6]} and 
G h = {e h ]A - e, A + e[} for ail A > 0 and for all e > 0. 
Theorem a.1. For each Bord set t" C X. and for each [-0% +oo]-val.ued Bord 
measurable function h on X satisfgin9 
lira liln sup/zta  (e h/t° l{>^t}c)') = 0. 
hI--ec 

(3.2) 
we bave 
(3.3) 
(3.4) 

for some 11 e [0, +ce[. Moreover. li,nl, t(e h/t° 1,-) ezists if 

sup {(A-g) limilfpâ°(FÂ, efY)}= sup {(-e) lilnsupHta(Fîeç}')}. 
A_>0,e>0 A>_0,e>0 
Pro@ Let Y be any Bord subset of X. Put 9 = e, G.e = Gî,«Y, F.« = Fî.eY 
for ail A > 0 and for all « > 0.  have 
limsup«(91/tly)  limstlp(91/tlF.) 
fo If x 
 ( - e) lira sup g [ ,«) 
for all  OE 0 and for all e > 0, and so 
(3.5) limsupg(g/tly)  sup {(-e) limsup(F,«)}. 



2910 HENRI COMMAN 

(3.7) 
Since 

Thus, in oder to prove (3.3) and (3.4), we have to prove that for SOlne/[ < oe, 
(3.6) lilns,,PPâ (gl/t«l,) i sup 
{x;«>o:g(x)_<M} 
For ail M  0, for ail N  N* and for ail 1 G j G N, we define 
Iç hae 
N 
1/t 
j=l 
 llI3X lims,,p l,  (glt 1FM..,) V liIllSllp « (gl/G l{g>M}y)" 

lira slip p (1/t ] FM,N,J ) (__ |illl 8/lp ]lta c' (&[,N,j)llg F,.. II, 
it follows from (3.7) that 
lira sup , (9 /t 1 ) 
(3.8)  max ]]91FM.]llimsuplz(F,,N.y) vlimsupl*(91/tl{e>t}m'). 
IjN ' 
Let AI  , N   in (3.8) and use (3.2) to obtain solne Alo  [0, [ such that 
(3.9) lilnsuplt«(gl/t«ly)  liminf lnax{I]glFM.m, lllilnsupp (F,NO)}. 
Thus, to obtain (3.6) it suffices to show 
lilll inf lnax { [91F,.. [1 linl 8up , (Mo,N.j) } 
(3.10)  8Up {(g(x)--)liln8tlp,oE(g(x),¢)}. 
If (3.10) does not hold, then there exists oe > 0 such that 
lira inf lnax { I[gl F»,,, H liln slip ,â (Mo,N,j) } 
Noe N- 
(3.11) > sup {(g(x) + p E) lilll sup t« 
- , 
Take 0 < o < oe/2 in (3.11) and obtain 
linl inf max {[]glFMo.m l linl slip Ira « (FMo.Nd) } 
(3.12) > sup {(g(x) + «o) lira sup " (G(.),«)}. 
But for ail 0  A  Mo and for ail N > Mo/«o we hae 
(3.13) (A + Eo) liInsup t l t'F 
, (a.)> lbu,., llil,p. t ,..) 
where jx is such that k e [(jx-1)ao/N, jxMo/N] (since [(jx-)Mo/', ix alo/N] c 
]A - «o, k + «oD- When A ranges over [0, klo], jx ranges over {j" 1  j  N}, and 
(3.13) implies 
O<<Mo 
(3.4) 2 max {llg,,..[limsup(F, NO)} 



CRITERIA FOR LARGE DEVIATIONS 2911 

for ail N > Mo/eo. Notice that ri)r ail N E N* a.nd for ail 1 _< j _< N, if 
then j = Jg() for some x  Y. Thus it suffices to consider A  {g(x) • 
Mo} in the L.H.S. of (3.14), that is, 
snp {(g(x) + eo)lin, supltL«(G9(),eo)} 
for all N > Mo/eo, which contradicts (3.12); it follows that (3.10), (3.6), and finally 
(3.3) and (3.4) hold. In the samc way that we obtainod (3.5), we have 
(3.15) liminf#L«(gi/t«ly)  su I) 
k0,e>0 
and the last assertion follows from (3.4). 
A localized version of Varadhan's theorem states that if X is regnlar an(l if 
satisfies a large deviation principle with t)owcrs (te,) and tight rate function .]. then 
(3.18) and (3.19) hold with 1 = J ([2], Exercise 4.3.11). Thc following corollary 
removes all the hypotheses on / and X. 
Corollary 3.2. Let 1 be a [0, +oe]-valued function on X satisfying 
(3.16) F  , limsup#L«(F)  supe -() 
(3.7) (re.»p. G  Ç, liIninfpL«(G) k suI) e-()). 
xG 
Tben, for each [-oe, +oe]-valu«d continaous function h on X satisfying (3.1), we 
(3.18) F  , liInsupltL«(«n/t"lF) 5 sup eh()« 
(3.19) (r«sp. G  Ç, lira inf ltL « («h/t 1) k sup «h()«-()). 
Proof. Suppose that (3.16) holds and (3.18) does hot hold for some [-, 
valued continuous function  on X satisfying (3.1). Since 
for all F G U by Theorem 3.1, there exists Fo G U, xo G Fo with h(xo) < OE, a.nd 
eo > 0 such that 
(e (°) eo)limsup/tL" F  

sup eh(x)e -l(x). 
xEF, h(x)< 

By (3.16) we have 
(e h(*°) - 0) sup e -l(*) > 
xE Fh( o) .o nF 
and so there exists x  Fh(o),o ç) F such that 
(e h(x°) -- ¢Eo)e -t() > sup eh(x)e-l(x. 

Since e h(*) _> e h(*°) - eo we obtain 

sup eh(*)e -t(x), 
xF,h(x)<oe 

xF,h(x)< c 

eh(**)e-l(**) > 

sup eh(*) e -t(x) 



2912 HENRI COMMAN 

with xl Ç F and h(Xl) < , whence the contradiction. Supt)ose now that (3.17) 
holds and (3.19) does hot hold for some [-, +]-valued (.oltillUOUS ftulctioi1 h 
on X satisf'ing (3.1). By (3.15), there exists C0 Ç Ç such that 
sup e()e -/() > lira inf « (e /t« 1o) 
xEGo,h(x)<oe 
 sup {(e h(z) - ) lira inf p.â  (G he(«  Go)}. 
{x6Go 
Thus. there exists .vo Ç Go with h(xo) < , and u > {} such that 
ch(*°)e -t(z°) > u + sup {(e h(x) - ¢) lira inf p.  «"¢),e  Go)} 
{xGo,e>O:h(x)<oe} 

and 1)y (3.17) 

«t(œe°)e-I(x°) > e t(x°) sup e -l(x), 
x- Gêh(xO) ,eO CIG 
which gives the contradiction.  
A direct consequence of Corollary 3.2 is that Varadhan's theorem can be stated 
verbatim for a general state space and with any function (in place of a tight rate 
flmction) I • X  [0. +] satisfying the large deviations lower and upper bounds: 
(3.21) limsupp,(F)  supe -() 5 supe -()  liminflt(G) 
xGF xGG 
for ail F Ç . G G Ç wi/h F C G: that is to say. A(h) exists and 
,X() = ,,p {() - («)} 
xX,h(x)<oe 
for ail [-OE,, +.]-valued continuous flmctions h on X satis6"ing (3.1). X will see 
wi/h Corollary 3.4 that it is possible to go fur/her in the generalization of Varadhan's 
theorem ot)taining/he saille coilclusiOlS with hypothesis weaker than (3.21). 
ecall that X is norlnal if and only if the following interpolation property holds: 
if f and g are real-valued respectively upper and lower seini-coiltinuous functions on 
.k" such tha/f  g, then there is a continuous fllllCtiOll  Oll X satis6"ing f 5 h  g. 
Theorem 3.3. Consider the followig statements: 
(i) A(h) exists for all h Ç Cb(X); 
(il) limsuppâ(F)  liminf p.â(G) for all F G . G G Ç with F C G. 
If X is normal. then (i)  (il). If (ii) holds, then (i) holds and moreover for each 
[-, +]-valued cont.inuous Iunct.ion  on X satisfying (a.1) we hat, e Ior some 

ai • [0, +[. 
(3.22) e h(h) = slip {()-e)7(Fî, e) } =- 
A>_0,e>0 
for all maps 7 " .7 tO ç -- [0.1] satisfging 
(3.23) 

{xX,e>O:eh() <_M} 

limsuppto(F) <_ -)(F) <_ 7(G) <_ lira inf 



CRITERIA FOR LARGE DEVIATIONS 2913 

for all F E., G E ç with F c G. 

Pro@ Suppose that (i) holds and X is normal. For each F ff  and G ff Ç with 
F C G, there exists h  Cb(.') such that 1F  h  1 G. Since 1F  e nh-n  
e-"l.x.kG V 1G for all   N, we obtain 
lira sup p (F) 5 inf e A[nh-n) 
 inf liminf{e - + #tg(G)}  liminf p,(G) 
and (il) holds. 
Suppose that (il) holds. Let h be a [-, +]-valued continuous fimction on X 
satisfying (3.1), and 7 " lU Ç + [0, 1] satis(ying (3.23). Put 9 = eh and let us use 
the saine notation as in the proof of Theorem 3.1 (with l = A-). For all A  0. for 
all e > 0 and for all 6 > 0 with  > e, we have by (3.23), 
lira inf item(9 /«)  lira inf 1,(9 /t 
 (- 6)7(G,6)  (A- 6)7(F,«). 

Thus 

and 

lira iaf p.to (91/to) >_ lira (,k - 5)3'(FA,e) 
_> (,x- e)7(F,) _> ()- 

liminf p, tâ(91/t') >_ sup 
A>_O,e>O 
> ,,p 
A>O,e>O 
In order to prove (3.22), we bave to prove that for some M < 
(3.24) limsupp,(g 1Ce)  sup {(e h() - «)(G,«)}. 
{ x X,e >O:eh(    I } 
Bv using (3.23), and in the saine way that we have obtained (3.10) in the proof of 
Threm 3.1, we find some M0  [0. [ such that to prove (3.24) it suffices to prove 
which is achieved exactly as in Theorem 3.1. 

The following corollary gives sufficient conditions much weaker than large de- 
viations with tight rate function in order to have the conclusions of Varadlmn's 
theorem; in fact, we will see in the next section (Corollary 4.2) that when X is 
normal, the condition (3.25) is also necessary. 

Corollary 3.4. Let 1 be a [0, +oo]-valued function on X satisfying 
(3.25) lira sup p,to (F) < sup e -t(x) < lira inf lâ ° (G) 
x6G 
(3.26) (resp. lira sup #tâ (F) 
for alI F  . G E ç with F c G. Then, A(h) ezists and 
(3.27) A(h) = sup {(x) - 
xX.h(z)<oc 



2914 HENRI COMMAN 

for all [-oc, +oc]-valued continuous functio. h on X satisfyi. (3.1). 
Pro@ Let h be a [-oc, +oc]-valued COl,tilmous fullction Oll X satisf.ving (3.1). If 
(3.25) holds, thon by Theoreln 3.3 (with 7(G) = supzec; e -t(z) for all G E Ç), A(h) 
exists and 
e A(h) ---- sup {(A -- e) sup e-l(z)}. 

Since for all A _> 0, e > 0 and x E Gî., 

(A -- e)e -l(z) <_ eh(z)e -l(z), 

we obtain 

and thns 
C A(h) _ Sllp eh(z)e -l(x). 
xEX,h(x)<oc 
For all x  X with h(x) < oc, and for all e > 0 we have 
(e h(x) -- ¢)e -l(z) <_ (e h(z) -- 6) sup 
YGêh() ,« 

which implies 

(eh(z) _ )e-l(:r) <_ e A(h), 
eh(z)e-l(z) <_ e A(h), 

and finally 
sup eh(z) e -(z)  e ri(h). 
xX.h(x)<oc 
Thus e h) = supex.h(z)< eh(X)e -t(z), which is equivalent to (3.27). If (3.26) 
holds, we conclude by applying Theorem 3.3 (with 3'(F) = supzeF e -(z) for ail 
F  9r), and replacing G h,.« bv. Fî.«, and G heh(), e bv. Fëh(), e in the above proof. [] 

Remark 3.5. Let F be the set of lnaps - : 9 r U Ç - [0.1] such that r(F) _< r(G) 
for all F  9 r, G  Ç with F C G. Define the narrow topology on F as the 
coarsest topology for which the lnappiugs 3' * 3'(}') are upper semi-contilmous for 
all Y E 9 r and lower semi-continuous for all Y  ç The net (#" (.1/t,)) can be seen 
as a net in F provided with the narrow topology, as well as a net iii [0, 
provided with the product topolog:y: Then. the implication (ii) => (i) in Theorem 
3.3 means that if (tttâ(.1/t«)) has a limit iii F, then (tttâ (-1/% )) has a limit in 
[0, oc[{h:heG(¥)}; lnoreover, the converse holds if X is normal. Of course, the 
limit in F when it exists is hot Ulfique: for each F  9 c and G  ç " defined bv 
-(F) = lira sup #tâ (F) and "(G) = lira inf ptâ (G) is an example, and "' defined bv 
")"(G) = 7(G) and 3/(F) = infa»F.aeç 7(G) is another oue. 



CRITERIA FOR LARGE DEVIATIONS 2915 

4. CR1TER1A OF A LARGE DEV1AT1ON PR1NCIPLE 

In this section, we investigate what has to t)e added to the existence of A(h) for 
ail h E Ch(X) (in other words, of the limit A(-) of (logltt«(e /t° ))iii ]-o% +oc[ ch(X)) 
in order to have large deviations. Of course, SOnle hyl)otheses on X are required to 
have sufficieatly continuous functions; so we sui)pose here that X as normal. In this 
case, by Theorenl 3.3 (alad Relnark 3.5) the existence of A(-) on Ch(X) as equiva- 
lent to the existence of a narrow set-theoretic lilaait 7 E F of (pt«), which as also 
equivalent to tlae existence of A(h) for ail [-oc, +oc]-vahled continuous finl«tions 
h on X satisfying tlae rail condition (3.1); nloreover, fi)r eacla such finlction h. the 
variational form of A(h) as given ila terlns of '7. In I)articulal, "7 can vary and it as 
essentially this flexibilitv which allows us to o|)taila in Theoren 4.1 lmcessary and 
suflïcient conditions, each of thenl corresponding to sonle tyi)e of infornmtion: a 
property of A(-) as a fun«tional iii (ai), a special variational fol'ail of A(') in (iii), 
a property of "7 iii (iV), and a property of the net (pt) ila (v) and (ri). It as 
worth noticing that in hoth formulations (flmctional (il) or sct-theoretic (iv)), the 
condition on the lilnit as thc saine: a continuitv property on ilacreasing nets. As 
corollaries, several basic results of the thcorv are strelagthened bv realoving the 
tightness or compactncss tayi)othesis. 

Theore,n 4.1. If X as no,wtal, then the follou,ing statements are equivalent: 
(i) (p) satisfies a large deviation principle wilh powers (t,). 
(ai) A(h) exisls for all h  Cb(X), and A(b) converges fo A(h) for each increa.s- 
in9 net (ha) in Ch(X) converging poinlu,ise lo h  Cb(X). 
(iii) A(h) exisls for all h  Cb(X), and A(h) = supex{h(x ) -/(.r)} for some 
function ! : X  [0. +oc] and for ail h  Cb(X). 
(iv) There as a map "7 : Ç -- [0.1] such that 
(a) lilnsuplt(F) _< "7(G) < linlinfpt(G) for all F  ., G  ç wzth 
FcG. 
(b) "7([.J, Gi) = linl'7(G) for each increasing net (Gi) i Ç. 
(v) A(h) exists for all h  C(X), ad for all F  .T, for all open covers {Gi : 
i  I} of F and for all e > O. there exists a finte subset {Gq, .... G u } C 
{Gi : i  I} such that 

(4.1) lilninfpt(/)- limsuppt( U Gi) < ¢. 

(va) There as a function l : X -- [0, +oe] such that 
(4.2) 
inf/(x)= sup {-liminftclogtz(G)} = su v {-limsuptclogp(G)} 
xF GÇ,GDF GÇ,GDF 

for all F  .. 
If (i) holds with rate function J. then the following properties hold: 

(4.3) inf J(x)= sup {-A(h)} for all F  ,'; 
xGF h.Cb(X),hlF=O 
(4.4) inf J(x) = sup {-A(h)} for ail G  ç, 
xG hCb ( X ),eh _lG 



2916 HENRI COMMAN 

where Che(X) is the set of [-oc, +oe[-valued bounded above cotiuous flmctions 
on X ; in particular, 

(4.5) J(x) = up {h(x)- A(h)} I« ll x e X- 

o 
(4.6) J = 1 yor ,,Il l" X -. [0. +] sti.yu,,9 (iii): 
(-1.7") e- J ( x ) = inf 
GEç,xEG 
]br all x e A. md for all 3" Ç  [0.1] sotisfying (iv): 
(4.8) ,l = 1 for all l" X  [0. +oe] satisfying (t,i). 
If morcover X is second cou,table, thon we tan replace "'net" bg "sequence" in (ii) 
Pro@ (i)  (iv) and (iii)  (ii) are «lear: (i)  (iii) bv Corollary 3.4 and so 
. V 
(i)  (ii) If (i) holds with rate flmction J. then for each F G . each open co er 
{Gi " i G I } of F and each e > 0, 

lira sul)l,â ° (F) < sup e -J < sul) e -J = sup sup e -J < sup lira inf pt« (Gi) ÷ 6, 

which implies (v). 

Suppose that (il) holds. We will prove that (i) holds. Let Che(X) be the set 
of I-et, +ec[-valued bounded al)ove continuous fimctions on X. By Theorem 3.3, 
A(h) exists in [-OE, +vc[ for ail h  Coe(X), and notice that 

A(h V k) = A(h) V A(k) 

for all k  Che(X); in particular. 

A(h V s) = A(h) V s 

for all s e [--ec,+ec[. Let (bi) be an increasing net in Che(X) converging to 
h  Che(X) with A(h) > -ec. For each real s < A(h) we have limA(hi V s) = 
A(h V s) = A(h), and so eventually A(h) > s, which shows that lira A(h) = A(h). 
Therefore, we can replace Ch(X) 1)y Che(X) in (ii). Let F e .T and h  Che(X) 
with hlF = 0. If A(h) > -oe, then A(hVs) = A(h) with (hVs)l F = 0 for ail 
s < A(h) A 0: if A(h) = -oe, then the sequence (A(h V --*))eN converges to --OE 
with (h V -n)l F = 0. Thus, 

inf e A(h) = inf e A(h), 
h Cba ( X ),h l oe=O h Cb ( X ),h l F =O 

and bv the interpolation property we have 

lira sup tt2 « (F) <_ inf e h(h) = inf e ri(h) 
h (7. Cba ( X ),h l F=O h (7.Cb ( X ),hl F =O 
(4.9) _< sup e h(h) <_ lira inf p (G) 
hGCba(X),eh_lG 

for ail F G .T, G G _G with F C G'. Define 

f() = 

inf c A(h) (= inf e't(h)) 
hGCb(A ),h(x)=O hGCb«(X),h(x)=O 



CRITEIRIA FOIR LAIRGE DEVIATIONS 2917 

for all x C A'. By (4.9), in order to prove (i) it sulïices to show that f is upper 
semi-continuous a.nd sa.tisfies 

(4.10) ,qU I) f(x) = inf e A(h) 
xE F hECb ( X ).h I oe =0 

for ail F C - and 

(4.11) su I) f(x) = sup e A(h) 
x:G hCb« ( X ),eh _lG 

for all G  Ç. We first show (4.10). Ch'arlv 

sui) f(x) <_ inf e A(h) 
xÇ F hCb( X),hlF=O 

for all F  -. SuI)I)ose that 
sup f(x) < e s < 

inf e A(h) 
hGCb( X ),hlF=O 

for some F  - and some real s. Then, for all .r  F there exists h« C Cb(X) «hich 
tan bc chosen negative such that ho(x) = 0 and 

(4.12) A(h«) < s < inf A(h). 
hECb(,\ ),hlF=0 

But 1F <_ e V¢ h with VeF h« bounded lower semi-continuous, and so there 
exists h  Ch(X) such that 1F N e h  e Ve h (in particular hIF = 0). Let I be 
the set of finite subsets of F ordered by inclusion, and bi = h A Vzei h« for all 
i  I, so that (bi)iCI is an increasing net in C(X) converging to h. Since A(hi)  
A(Ve h) = supzei A(hz) < s for ail i  I. we obtain limA(hi) = A(h)  s, which 
contradicts (4.12). Thus (4.10) holds.  now prove (4.11). Bv the interpolation 
property (between 1{«} and 1GE) ve have clearly 
sup f(x)  sup e A(h) 
xÇG hCba( X).ehlG 

for ail G  {ï. Suppose 

sup f(x) < sup e A(h) 
X _G h_Ctm (X) ,e h  1 G 

for some G  {ï. Then, for ail x  G there exists ha  Cb(X) with ha(x) = 0 such 
that 

(4.13) sup A(h) < s < A(ha) 
xG 

for some ha  Cba(X) with e ha <_ 1, and some real s. Let I be the set of filfite 
subsets of G ordered by inclusion, and bi = b A /œei hx for all i  I, so that 
(h)ie, is an increasing net in Cba(X) converging to hŒE. Then 

A(h) = limA(hi) <_ limA( V h)= lim(supA(h)) <_ s, 
xi xEi 

which contradicts (4.13). Thus (4.11) holds. It remains to show that f is upper 
semi-continuous. By (4.9), (4.10), (4.11), and since 
f(x) = inf e () _ < ilf sup e A(h) 
hÇCba(X),h(x)=O GD{x} hCb«(X),e  <_IG 



2918 HENRI COMMAN 

for ail x E X. by Lennna 2.1 il suflïces to prove that 
(4.14) f(x) = illf e A(h) >_ |nf slip e A(h) 
hECba (X),h(œe)=0 GD {œe} hECba (X),e h 
for all :r E N. Suppose that (4.14) does not |iold for solne :r  X. Then. there 
exists h«  Cb,(X) wit.h hx(x) = 0, and oe > 0 such that 
(4.15) eA(h) + u < |nf sut) 
GD{x} hCb«(X},eh_lci 
I3y (4.9) and Theoreni 3.3 (with 7(G) = SllPheCba(.\-),eh<_lci eh(h)), we have 
(4.16) e A(h) = sui) {(A - ¢) slip eh(h)}. 
.X>O,e >O h_ Cba ( X ),e h (1 
Take A = 1 alid 0 < e < ,, in (4.16), and obtain by (4.15), 
mil» c A(h) < |nf sui» 
hCba ( X ),e h (1Gh x GD { œe } hECba ( X ),e h  1 ci 
h, 
wit|l x  Gi., wh|ch gives the contradiction. Thus (4.14) holds and f is tlpper 
Selni-cont illllOllS. 
We have proved (i) = (ii), and that when (i) holds with rate flinction J, then 
(4.3) and (4.4) hold (by the miiqlleness of a rate flnlction on regular spaces); since 
A( - ()) = ,(,.) - (). 
(4.5) follows froln (4.3). 
Suppose that (iii) holds with l" X --, [0, +oc]. Then, obviouslv (il) and so (i) 
hold; let J be the associated rate filllctioll. By Corollary 3.4, 
(4.17") e A(h) --- slip ehe -J 

for ail h  Cb(X). and so 
(4.18) 

X 

Slip h--J = sup h-I 
X X 

for ail h  Ch(X). Clearly, for each h  Cb(X) there exists a real s such that 
Slip «he--J = sup ffhVse-J 

and 

and so by (4.18), 
(4.19) 

X X 

slip ehŒ -l --- Sllp ¢hVse-l, 

X X 

SUp c e- J = Slip eh e -I. 
X X 
For any G  Ç choose an increasing net (h) in Cb(X) such that slip/e h` = 1, 
and obtain by (4.19), 
slip e - J = Slip e -I. 
G G 
o o 
Since supG e -I - supG e -I by Lemnla 2.1, we llave supGe -d = supGe -I for ail 
o O 
G  Ç. Since e -J and e - are upper selni-continuous, we have J = 1 by Lenmla 
O 
2.1. Thus, if (i) holds with rate fUllctiol ,J, then J = 1 for all 1 • X 
satisf,ving (iii) and (4.6) holds. 



CRITERIA FOR LARGE DEVIATIONS 2919 

Suppose that (iv) holds with 7 " {7 --, [0, 11. Define 7(F) = infa»F 7(G) for ail 
F E f', and notice that 7 is increasing on f', sat.isfies 7(F) <_ 7(G) for all F E f', 
G  Ç with F C G, and by (a), 

(4.20) -( [_J a)<_ S,lp-r(aj) 
I<j<_N I<_j<_N 

for each finite family {Gj}I<j<N 
h _ Che(X) and 
(4.21) e A(h) =- 

(4.22) 

C ç Bv Theorem (3.3), A(h) exists for all 

We will show t.hat (il) holds. Let. (h)e bc an increasing net in Ch(X) converging 
to h  Ch(X), and suppose that A(h) > supe A(h). By (4.21) and (4.22), there 
exists h0 > 0 and e0 > 0 such that 
(Ao - eo)7(G,« o) > sui) sui) {(£ - e)7(Fî,) } 
iI 
iI 

and thus 
h 
7EE(Gîo,« o) > sup 7EE(Fî,e o) > sup'y(GAo,eo). 
iI iEI 
Let » be the set of finite subsets of I ordered by inclusion, and obtain by (4.20), 
h 
(4.23) Vil e , ?(Co,«o) > sup?( ' 
G,«). 

But G h h 
ao,«o C sup¢e Uie¢G)o,eo, and the condition (b) contradicts (4.23). It 
follows that A(h) = supie/A(bi), that is, (ii) and so (i) hold: let J be the associated 
rate function. We now prove that (4.7) holds. Let G  ç and h  Che(X) be such 
that e h _< 1«. For all x  X and e > 0 with e h(x) > e, we bave 

and 

Thus, 
(4.24) 

sup 

Fêh(),« C G 

(eh(x) _ 

{(eh(:r)--g)'7(Fêh,=),e) } <_ sup 
{xX :,e h(=) >e},e>0 

<_ 

and since if e A(h) > O, 
(4.25) e A(h) = 

p {(eh() _ 
{xX;e h(x)>},>O 

by (4.24) and (4.25) we obtain 
(4.26) sup 
hCba(X),ehlG 

e ri(h) _< 7(G). 



2920 HENRI COMMAN 

Sul)pose t hat 
sup e A(h) + ve < 
hECba( X ),e h lG 
for some u > 0. Bv taking A = 1 and 0 < e0 < u/2 in (4.22) we obtain 
(4.27) ?(Feo) + u/2 < ff(G) 
for all h  Cb.(X) such that e h  1G. Let (hi) be an increasing net in Cb.(X) such 
that sut) / e'h' = 1G, and let p be the set of finite subsets of I ordered 1)v inclusion 
bi 
Then (ieZ G,eo)e is an increasing net in Ç such that 

By (4.27) we bave 

and by (4.20), 

ri e ,, .,/(c,î:«o) + ./2 < ,,:(#* 
_ ,,«o) + ,/'2. < .,/(a), 

and by (4.4) 

sup e -a = -(G), 
G 
which gives (4.7) by upper senfi-continuity of e -d. 
Suppose that {v) holds. Bv Theorem 3.3. 
c A(h) .... sup {(A ¢)liminfpL"(Gî,e)} sui, {(A e) limsupp>(F,e) }h 
A0,e>0 AO,e>O 
for all h  Cb(X). Let (h)e« be an increasing net in Cb(X) converging to h  Cb(X). 
 will prove that limA(hi) = A(h). If A(h) > supie A(hi), then there exists 
Ao  0, o > 0 and oe > 0 such that 
(4.) 
(Ao eo)liminf h 
- (G,«o) > sup sup {(A-e)limsupp(Fî')}+u 
iÇI AO,e>O 
iEI AO,e>O 
iI 
Take eo < e < eo + u/2 in (4.29) and obtain 
(4.30) (Ao eo) liminf  Xo,«o/ > (Ao - eo)suplimsupt'Gh't o,«)" + /2. 
i6I 
Put F = F h Gi 
xo,eo' = Gxo,« for all i  I, and notice that F C ietGi" Since 
O 
FD G h 
xo.eo we obtain by (4.0), 
O 
lira inf 
iI 



CRITERIA FOR LARGE DEVIATIONS 2921 

alld so 
liminfPâ(l) > lilnsut'#â( U Gij)+ t/2, 
for ail finite subsets {G., " 1  j  N} C {G " i e I}, which contradicts (4.1). 
Thus limA(hd = A(5), tiret is, (il) and so (i) hold. 
It remains to prove (i)  (ri), (4.8) and the last assertion. Supl)ose that (i) 
holds with rate function .1. Put f = e -a and let F  Y. By (4.3) we have 
sup f(z) = inf e A(h)  inf lintinfp'(G)  inf limsut)l,(G). 
x F hCb ( X ),hI F=O Gç,(IDF GÇ,GD F 
Suppose that 
inf e A(h) < inf liln sup tt  
hCb( X ),hlF=O GÇ,GDF 
Then. there exists  > 0 and hF  Ch(X with hl F = 0 su«h that 
e A(h) +  < inf liln sup p 
GÇ,GDF 

(4.31) 
Since 
(4.32) 

e A(hF) sup { (c hE(z) - 
by Theorem 3.3. by taking e = e0 <  iii (4.32) we obtain by (4.31), 
s t(F(, eo)} ) < inf limsupl,k (G). 
(4.33) sup {e h(x) liln up Pa 
{ x X ;h F( x) <  } , GÇ.GD F 
h 
Since hF(X) = 0 for all • G F, we have F C G.eo 
liln sup p ,hu  < lim sup p. 
,,« _ (Ft,«) < 

hF 
C F[,«o and (4.33) ilnplies 
inf lira sup pâ (G) 
GÇ,GDF 

By combilfing Theorem 4.1 with Corollary 3.4 and Lemma 2.1, we obtain in the 
following corollary necessary and suflïcient conditions in order that a large deviation 
o 
principle occurs with rate function the lower regularization 1 of a given flmction 
l" X --, [0, +oo]. Notice that by Proposition 2.3 and (i) ¢ (ii) in Çorollary 4.2. 
the infilnum of the set of [0, +oo]-valued functions I on X satisfying (3.25) coincides 
o 
with t.he lower regularization l of each its elements. The equivalence (i) = (iii) in 

and the contradiction. We have shown that 
sup f(x) = inf lira inf pta° (G) = inf lira sup pta (G) for all F • 
xF GÇ,GDF GÇ,GDF 
which is equivalent to (4.2) with l = J. and so (ri) holds. 
If (ri) holds with l-X -- [0, +cci, then (4.2) ilnplies 
limsup #C(F) <_ supe - <_ liminf pâ° (G) 
F 
for all F • .T, G • Ç with F C G. By Proposition 2.3. (i) holds with rate ftlllCtiOll 
o 
l and (4.8) holds. 
If moreover X is second countable, then it is well knowll that for any familv 
{bi • i • I} of lower selni-continuous functions on X there exists a countable subset 
I0 C I such that supie bi = supie o bi. It is easy to sec iii the above proof that this 
property allows us to replace "net" by "sequence" in (il) (resp. (iv)), and "open 
covers" by "countable open covers" in (v). [] 



2922 HENRI ÇOMMAN 

Corollary 4.2 was known when 1 is a tight rate flmction ([2], Theorem 4.4.13): here 
there is no hypothesis on !. 
Corollary 4.2. Suppose that X is normal, and let I be a [0, + ]-valued function 
on X. Then. the following statements are equivalent: 
(i) (#a) satisfies a large deviation principle with powers (fa) and rate function 
o 
l. 

(ii) 

limsuppL " (F) _< sup e -(*) < lira lllft, a ((7) 
xEG 
for all F  , G  çwith F c G. 
(iii) A(b) exists and 
A(/,) = stlp{/,(x)- l(a')} for ail l, C Cb(X). 
xEX 
(iv) A(h) exists and 
A() = ,,p {h(«)-()} 
xX,h(x)< 
or all [-, +oe]-vol,,ed «o,tin,,ous unçtios  o X sati&ig (3.1). 
Pro«¢ (ii)  (iv)  (iii) by Corollary 3.4. (iii)  (i) by Theorem 4.1, and 
(i)  (ii) since supxeG e -(*) = sup,eG e -(*) for ail G  Ç by Lemnm 2.1.  
Recall that (Po) is said to be exponentially tight with respect to {ta) if for all 
g > 0 there is a compact set h C X such that lim supp (XkK) < «. Brvc's 
theorem ([2], Theorem 4.4.2) stores that if A(b) exists for all   Cb(X) and if 
(p) is exponentially tight with respect, to (t), then {p) satisfies a large deviation 
principle with powers (); moreover, the (necessarily tight) rate function satisfies 
(4.5). The following Corollary 4.a shows that the first conclusion holds under a 
hypothesis clearly weaker thon exponential tightness. Moreover, Theorem 4.1 stores 
that without anv tightness hypothesis, a rate function for (pâ) always satisfies 
(4.5). 
Corollary 4.3. Suppose t5at X is normal. If A(h) ests for all h  Ch(X), and 
gE for all open covers {Gi • i  I} of X. for all e > O. there exists a finite subset 
{«,, .... « } ç {«.  e } « a 
(4.34) limsupl,(Xk  Gi,) < g, 
t5e (I') satisfies a large deviation pri,,çiple with powers (). 
Pro4 Let F  , {G • i  I} be an open cover of F and e > 0. Then. 0e G O 
XkF is an open cover of X, and so there exists a finite subset {Çh,----Çi } C 
{Çi " i  I} such that 
linlsup#â(Xk(  Gi, UXk )) < g. 
IjN 

Thu8, 

lira sup #t (F \ U 
and bv Theorem 4.1, the conclusion holds. 



CRITERIA FOR LARGE I)EVIATIONS 2923 

Corollary 4.4. If X is 'normal and (lttâ ) satisfies (4.34), then (#,) bas a subnet 
(#) satisfyi.ng a large dew;ation principle with powers (t). 
Proof. Define A(h) = logp(e t/t) for ail h E Ch(X). Then. (A,(-)) is a net in 
the compact space [-et, +cci c(X) (with the product topology), and so there is a 
subnet (A(-)) converging to some limit A'(-). The result follows from Corollary 
4.3 applied to (/t ). [] 
Remark 4.5. (i) ** (iv) in Theorem 4.1 was known when 3' is a sup-preserving ca- 
pacity in the O'Brien sense, i.e., 7((7) 
Ç, (lç) = inf{7(G) : G D Iç.G Ç Ç} for all compact Iç c X, and -), satisfies (2.1) 
of Lemma 2.1 ([3]). Thus, Thcorem 4.1 removes the SUl)-preserving as well as the 
capacity conditions of-y (notice the difference between (iv) in Theorem 4.1 and (ii) 
in Proposition 2.3). In the spirit of Remark 3.5. this means that (It,) satisfies a large 
deviation principle with powers (te) if and only if (pâo) has a narrow set-theoretic 
limit in the set {7 G F : lim/(G) = ")(Ui Ci) for all increaing nets (Gi) in Ç}. 
Remark 4.6. When X is Polish and (p) 
Bell ([1], Theorem 2.1) implies the equivalen«e of the following statements: 
(i') (l) satisfies a large deviation prin«iple with powers (l/n) and tight rate 
fimction: 
(il') A(h) exists for all h Ç Ch(X), and infr A(hr) = A(h) for each decreasing 
sequence (bru) in Ch(X) converging to h  Ch(X). 
The equivalence (i) = (il) together with the last assertion in Theorem 4.1 tan be 
seen as a free tightness analogue of that, by replacing "decreasing" by 'increasing" 
in (ii'), and removing "tight'" in (i'). 
Remark 4.7. The relation (4.2) in Theorem 4.1 generalizes a well-known expression 
of a rate function J for (#tâ), obtained with 1 = J and F ranging over all singletons 
([2], Theorem 4.1.18). 

l EFERENCES 

1. W. Bryc and H. Bell. Variational representations of Varadhan functionals, Proc. Amer. Math. 
Soc., 129 (2001), No. 7, pp. 2119-2125. MR 2002b:60040 
2. A. Dembo and O. Zeitouni. Large deviations techniques and applications, Second edition. 
Springer-Verlag, New York, 1998. MR 99d:60030 
3. G. L. O'Brien and W. Verwaat. Capacities, large deviations and loglog laws. Stable Processes 
and Related Topics (Ithaca, NY. 1990), pp. 43-83. Progr. Probab. 25. Birkhiuser, Boston, 
MA, 1991. MR 92k:60007 

DEPARTMENT OF I,IATHEMATICS, UNIVERSIT5 OF SANTI.GO OF CH1LE. BERNARDO O'HIGG1NS 
3363. SANTIAGO, CHILE 
E-mail address: hcomma_n@usach, cl 



TttANSACTIONS OF THE 
AMEttlCAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2925-2948 
S 0002-9947(03)03256-2 
Article electronically published on February 25, 2003 

INTEGRATION BY PARTS FORMULAS INVOLVING 
GENERALIZED FOURIER-FEYNMAN TRANSFORMS 
ON FUNCTION SPACE 

SEUNG JUN CHANG, JAE GIL ('ttOl, AND DAVID SKOUG 

ABSTRACT. In an upcoming paper. Chang and Skoug used a generalized Brow- 
nian motion process to define a generalized analytic Feynman integral and a 
generMized analytic Fourier-Feynman transform. In this paper we establish 
several integration by parts formulas involving generalized Feynman integrals, 
generalized Fourier-Feynlnan transforms, and t|le first variation of funclionals 
of the form F(x) ---- f((¢l,X),...,(¢n,X)) where (¢,x) don(,tes the t'aley- 
Wiener-Zygmund stochastic integral fo r c( t )dx( t ). 

1. INTRODICTION 

In [11], Park and Skoug, working in the sclting of one-parameter \Viener sl)a.ce 
C0[0. T] established several integra.ti(m t)y parts formulas involving analytic Feyn- 
man integrals, Fourier-Feynnmn transfonns, and the first variation of flmctionals 
of the form 

(1.1) 

where (.x) denotes the Paley-Wiener-Zygmund sto«hastic integral  
In this paper, we also studv flmctionals of the form (1.1) but with :r in a very 
general function spa«e Ço,b[O,T] rather than in the Wiener space Ç0[0. T]. The 
Wiener process used in [11] is flee of drifl and is stationary in rime while t.he 
stochastic process used in this pa.per is nonstationary in rime, is subject to a drifl 
a(t), and can be used to explain tlw position of the Ornstein-Uhlenbeck process 
in an external force field [10]. It turns out, as noted in temark 3.1 bclow, that 
including a drifl terre a (t) ma.kes est.ablishing various integration bv parts fonnulas 
for Fourier-Feynumn transforms very diflïcult. 
By choosing a(t) = 0 and b(t) = t oll [0, T], the function space C,b[O, T] rednces 
to the Wiener space C0[0, T], and so the results in [11] are i,nmediate corollaries of 
the results in this paper. For related work see [3], [4], and [6]. 

Received by the editors September 6, 2002 and, in revised form, November 15. 2002. 
2000 Mathematies Subject Classification. Primary 60J65, 28C20. 
Key words and phrases. Generalized Brownian motion proeess, generalized analytic Feynman 
int.egral, generalized analytic Fourier-Feynman transform, first variation, Cameron-Storviek type 
theorem. 
The present research was conducted by the research fund of Dankook University in 2000. 

(2003 American Mathematical Societ_ 



2926 SEUNG JUN CHANG, JAE GIL CHOI. AND DAVID SKOUG 

2. DEFINITIONS AND PRELIMINARIES 
In this section we list the appropriate preliminaries and definitions ri'oto [5] that 
are needed to establish our parts forinulas in Sections 3, 4 and 5 below. 
Let D = [0, T] and let (t, B. P) be a probability measure space. A real-valued 
stochastic process Y on (-, B, P) and D is called a generalized Brownian motion 
process if l'(0, w) = 0 ahnost everywhere and for 0 = t0 < t < --- < t, _< T, the 
n-dimensional random vector (Y(tl, w),-.- , Y(t,,w)) is normally distributed with 
the density function 
Iç({:,,ï) = ((2r) ' H(b(tj)-b(tj_,))) -/2 
(2.1) 
{ 1- ((qj-a(tj))-('tlj-l-a(tj-1)))2} 
-exp - b(t)-b(t_) 
where -q = (q,-.-, q,,), q0 -- 0, ï= (t,.-.,t,), a(t) is an absolutely continuous 
real-valued function on [0,] with a(0) = 0, a'(t)  L[0, T], and b(t) is a strictly 
increasing, continuouslv diffcrentiable real-valued function with b(0) = 0 and b(t) > 
0 for each t G [0, T]. 
As explained in [13, pp. 18-20], Y induces a probability measure p on the measur- 
able space (]D 13 D where ]I Dis the space of all real-valued functions x(t), t G D, 
and D iS the smallest a-algebra of subsets of ]D with respect to which all the 
coordinate evaluation maps et(x) = x(t) defined on ]Dare measurable. The triple 
(]D BD p) is a probability measure space. This measure space is called the func- 
tion space induced by the generalized Brownian motion process Y deternfined by 
a(-) and b(-). 
Ve note that the generalized Brownian motion process Y determined by a(-) and 
b(-) is a Gaussian process with mean function a(t) and covariance function r(s, t) = 
min{b(s),b(t)}. By Thcorem 14.2, [13, p. 18ï], the probability measure p induced 
by Y, taking a separable version, is supported by C«,b[O, T] (which is equivalent 
to the Banach space of continuous functions x on [0, T] with x(0) = 0 under the 
sup norm). Hence (Ca.bI0, T], B(C«,b[0, T]), p) is the function space induced by 
where B(C«.b[0, T]) is the Borel a-algebra of C«,b[O, T]. 
A subset/3 of Ca,o[0, T] is said to be scale-invariam nmasurable [9] provided pB 
is B(C«,b[0, T])-measurable for all p > 0, and a scale-invariant measurable set N is 
said to be a scale-invariant null set provided p(pN) = 0 for all p > 0. A property 
that holds except on a scale-invariant null set is said to hold scale-invariant ahnost 
everywhere (s-a.e.). 
Let Lâ,b[O, T] be the Hilbert space of functions on [0, T] that are Lebesgue mea- 
surable and square integrable with respect to the Lebesgue-Stielt.jes measures on 
[0, T] induced by a(-) and b(-); i.e., 
L«,[O,T] = v" v2(s)db(s) <  and v2(s)dlal(s) < 
where lai(t) denotes the total variation of the function a on the interval [0, ri. 
2 
For u, v G L a,b[O, T], let 
T 
(2.3) (u, v)«,b = I u(t)v(t)d[b(t) + lai(t)]. 
J0 



INTEG1RATION BY PARTS FO1RMULAS ON FUNCTION SPACE 2927 

Then (-, .). is an inner product on L2,b[O. T] and I[ull,b = v/(u, u),b is a norm on 
L,b[O,T 1. In partioflar, note that Ilu[I«,b = 0 if and only if u(t)= 0 a.e. on [O,T]. 
N, rthermore, (L,b[O,T], . l],b)is a separable Hilbert space. 
Let {j}= be a complete orthogonal set of real-valued flmctions of bounded 
variation on [0, T] such that 
{0. 
(¢»¢«).b = . j = ., 
and for each v  Lâ,b[O. T], let 
(e.4) ,(t) = (,, ¢),.b¢(t) 
for  = 1,2,.-.. Then for each v  Lâ,b[O, T], the Paley-Wicner-Zygmund (PWZ) 
stochastic integral (v, x) is defined by the formula 
(.5) (,.) im ff 
= 
for all x  C«.b[O. T] for which the limit exists: one tan show that for each v 
c.d0.rl, the PWZ integral (v,x} exists for p-a.e, x 
g denote the function space integral of a B(Ca.b[O. T])-measurablc fimctional 
F by 
Jc 
whenever the integral exists. 
 are now ready to state the definition of the generalized analytic Feymnan 
integral. 
Definition 2.1. Let C denote thc complex numbers. Let C+ = {A  C • ReA > 0} 
and Ç+ = {A e C"  ¢ 0 and Rea  0}. Let F" C,b[O. T]  C be such that for 
each A > 0, the function space integral 
Jc 
..[0.TI 
exists for ail A > 0. If there exists a function J*(A) analytic in C+ such that 
J*(A) = J(A) for all A > 0, then J*(A) is defined to be the analytic fimction space 
integral of F over C.b[0, T] with parmncter A, and for   C+ we write 
(e.7) «' [1  _Z' [(x)] = *(a). 
Let q ¢ 0 be a real number and let F be a functional such that E  [F] exists for 
M1 A  C+. If the following limit exists, we call it the generalized analytic Feymnan 
integral of F with parmneter q and we write 
(.S) «'[1  «2"[()l = im 
--iq 
where A approaches -iq through lues in C+. 
Next (see [5], [7], [1], [8], and [6]) we state the definition of the generalized 
analytic Fourier-Feynnmn transform (GFFT). 



2928 SEUNG .IUN CHANG. JAE GIL CHOI, AND DAVID SKOUG 

Definition 2.2. For A • C+ and y • G,b[0, T], let 
(2.9) Ta(F)(9) = En'[F(9 + x)]. 
For p • (1.2], we define the Lp analvtic GFFT, Tq(p:F) of F, by the formula 
( e c+), 
(2.10 / Tqp:F)(9)= lin, T(F)(9) 
if it exists; i.e., for each p > 0. 
k" define the L analvtic GFFT. Tq(l: F) of F, bv the 

where 1/p + 1/p' = 1. 
fiwmula (A • C+) 

(2.11) Tq(l: F)(9) = lira Ta(F)(9) 
A--,-- iq 

if it exists. 

We note that flw l <_ p <_ '2. Tq(p: F) is only defined as s-a.e. \Ve also note that 
if Tq(p: F) exists and if F  G, then Tq(p: G) exists and Tq(p: G)  Tq(p: F). 
Next we give the definition of the first variation of a fimctional F on C«,b[O. T] 
tbllowed by a very fimdaln«ntal Cameron-Storvick type theorem [2] which was es- 
tablished in [5. Theorem 3.5]. 
Definition 2.3. Let F be a B(C«,b[O, T])-measurable fu,lctional on Ce.bi0, T] and 
let w  C«.b[O, T]. Then 
(2.2) aF(zl-a,) = F(.r + 
h=0 
(if it exists) is called the first variation of F. 
Throughout this paper, when working with 3F(z]w), we will alwavs require w 
to be an element of .4 where 
(2.13) A = {u, e C«,b[O,T] " w(t) = z(s)db(s) Ol- some z e L]b[O.T]}. 
Note that for F(z) of the form (1.1), aF(x[w) acts like a directional derivative 
in the direction of u,. For example, if f(ul,U2) = exp{3u + 4u2} and F(z) = 
((, ), (,.)), thl, 
6F(z[w) = [a(a, w) + 4(a2. w)] exp{a(a, z) + 4(a2.z)} 
= (c, )I, ((, ), (, ')) + (, ,,') f((» «), (, «)). 
The following notation is used throughout the paper: 

(2.14) 

alld 

(2.15) 

(u2,b ') = u2(t)b'(t)at = 

for tt • L2.b[O, T]. Furt.hermore for ail A • C+, v is 
real part. 

u2(t)db(t) 
alwavs chosen to bave positive 



INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2929 

Theorem 2.1. Let z E L2,b[O, T] be given and fort  [0, TI, let w(t) = .1 z(s)db(s). 
For each p > O, let F(px) be p-integrable on Ce,bi0, T] and let F(px) hat,e a first 
variation 5F(px]pw) for ail x  C«,b[O, T] su«h thal for so'me positive fun«tio,t q(p). 
in t« ollo'wig euation «ist, t« t tird on« aIso «ists, d «qulit olds: 
(2.16) nf [SF(lw) ] - anf 
_ =-q [F(.)<z,«>] - (-@(=,a  [(x)]. 
In fact. for each A  C+, the above conclusions also hold for analgtic fwwtion space 
integrals, 
ç finish this section by stating a very fimdamental integration tbrmula fin the 
function stmce C«,b[O, T]. 
Let {1,"" ,«b} be an orthonormal set of funçti«ms from (L,b[O,], 
and for j  {1,--- ,zt} let 

T 
(2.18) 4j = (aj,a') =/o aj(t)da(t) 

and 
(2.19) Bj  (a,b')=  a(t)db(t). 
Note that Bi > 0 for each j ( { 1, 2,--. , z}, while for each j, A may be positive, 
negative or zero. 
Let f-   be Lebesgue mesurable, and le F(x) = f({al,X},.--, 
Then 
dc 
(2.20) Ç ,, )_} { (uJ-Aj)2}du, dan =  Y(.1,---..)p .... 
_ . ' =l 
in the sense that if either side exists, both sides exist and equality holds. 

3. INTEGRATION B'T PARTS FORMULAS ON FUNCTION SPACE 

Let n be a positive integer (fixed throughout this paper) and let {01, • - • , t't.} be 
an orthonorlllal set of ftlnctiolls fronl (Lâ,b[O, T], I1" lit,b)- Let m be a nommgative 
integer. Then for 1 < p < oe, let B(p; m) be the space of all functionals of the 
form (1.1) for s-a.e, x  C«,b[O,T] where all of the kth-ordel" partial derivatives 
fjl,...,j(Ul,''" ,Un) = fj,....,j(ff) of f: '-  are continuous and in LP( ") fOl" 
k ( {0,1,--. ,rn} and each j ( {1..--,n}. Also, let B(cx:m) be the space of all 
flnctionals of the forln (1.1) for s-a.e, x ( C«.b[0, T] where for k = 0.1,.-- ,m. 
all of the kth-order partial derivatives fil,... ,j» (ff) of f are in C0(), the spa«e of 
bounded continuous functions on  that Valfish at infinity. 
Out first lelnma follows directlv from the defilfitions of 5F(x]w) and B(p: m). 



2930 SEUNG JUN CHANG, JAE GIL CHOI, AND DAVID SKOUO 

Lemlna 3.1. Let 1 <_ p < oc be given, let m bc a positive irteger, let F 6/(p: m) 
be given by equatio (1.1) and let w be an element of A. Then 
(3.1) F(xlw) = (oj,w)fj((Ol,£,... , 
j=l 
for s-a.e, x  Ca,[O, T]. Furthermore. as a function of x. 6F(-]w)  (p; m - 1). 
Lemma 3.2. Let p, m ad F be as ir Lemma 3.1. Let z  Lâ,[O. T] be 9iven. and 
for t e [0. T], let w(t) = fO z(s)db(s). Let G e B(p';m) be given by 
(3.2) G(«) = g((ol,x),-'-, 
o .-a.«.. z Ca.[0. T]. fi,, (') = F()G() o 
R  (1: m), azd as a flmction ofJ'. 5R(.Iw)  (1: m - 1). 
Proof. Let r(Ul,... ,Un) = f(ul,"" ,Un)9(Ul,''" ,Un). Then (x) 
--- , (a, x)) is an clement of (1: m) since all of the kth-order partial derivatives of 
r are contimous and in LJ() for k = 0.1,--- , m. Applying Lennna 3.1 we obtain 
that R(x[w), as a fimction of x, belongs to B(I: m - 1). 
Remark 3.1. Let F, G and R be as in Lemma 3.2 above. In evaluating 
z[n-x)], a,d E[5n-I,)] f  > 0. th 
(3.3) H(A;u,...,u)H(A:ff)=exp{-£(uJ-A)2} 
=1 
occurs, where Ai and Bj are given by equations (2.18) and (2.19) above. Clearly. 
for  > 0, H(A: ff)  1 for all ff G  since Bi > 0 for ail j = 1,-.-, n. But for 
 G Ç+, ]H(A;ff)] is hot necessarilv bomded by 1. Note that for each  G Ç+, 
 = c +id with c  ]d]  0. Hence, for each  G 
H(A;ff) =exp{ - (uj - Aj)29= 2Bj } 
= exp {  [(c2-d2+2«di)u-2(«+di)Ajuj+A] }- , 

and so 

(3.5) H(,: ff)] = exp { - - [(c2 - d2) -- 2-cAjuj + A'] } 
3 =1 -2Bi " 
Note that for , C C+, the case we consider throughout Section 3, Re() = 
c > ]d = hn()[  0. which implies that c 2 -d 2 > 0. Hence. for each A G C+, 
H(A; ff), as a function of ff, is an element of LV( ) for all p  [1. +]: in fact, 
H(A: ff) also belongs to C0(). These observations are critical to the development 
of the integration by parts formulas throughout Section 3. 
In Sections 4 and 5 below we consider the case where A = -iq  +-C+. In this 
ce,==«+idwith«==]d,. Hen«e, forA=-iq, qeN-{O}. 
c  - d 2 = O, and so 
(3.6) [H(-iq:ff), = exp{  [Ajuj - A] } 
=1 2B " 



INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2931 

which is not necessarily ill LP(IR n) for any p E [1, +oc]. Thus, il Sections 4 and 5 
we will need to put additional restrictions Oll the functionals F and G in order to 
obtain the corresponding parts formulas iuvolving Fourier-Fevnmm transforms. 

Remark 3.2. Note that in the setting of [11], a(t) = 0 and b(t) = t on [0, T] and 
so Ai = (ctj,a') = 0 and Bi = (ct,b') = 1 for all j e {1,2,-.-,r}. Hence, for all 

IH(A:/7)I= exp - u =exp 

u <1. 
2 -- 
j=l 

Theorem 3.3. Let z e L,b[O.T ] be given and fort e [0. T], let w(t) = f z(s)db(s). 
Let p, m, F and G be as in Lemma 3.2. Then for all A  C+, 

(3.7) 

where x/ï is chose to have positive real part. 

Proof. First define R(x) = F(x)G(x) an, t let 

r(a,-.-,,) : f(u,.---.an)g(,q,"". ,). 
Then by Lemma 3.2, R E B(1; m) and (R(.[w) /3(1: m- 1). Furthermore ail of the 
kth-order partial derivatives of r are contimous and lu LI(iR n) for k = 0, 1.--. , m. 
Hence, R(px) is p-integrable on C,,b[O, T] for each p > 0. In addition, for s-a.e. 
x e C.b[O. T], 
5R(x[-w) = F(x)SG(xlw ) + 5F(x[w)G(x) 
= f((Ol,X),''" , (Otn,X)(Otj,W)j((OI,X),''" , 
(3.8) =1 
+ g((Ol,X),"" , (On,X)) '(Oj,w)fj((Ol,«),'", (an,x)). 
j=l 

But for ail u e L.b[O, T], 

(3.9) 

Jô0 T 

.()d,() 

u(s)z(s)db(s) 
T 
< fo Iu(s)z(s)ld[b(s) + lai(s)] 

In particular, since {eq,---,cn} are orthonormal, I(» ')1 < Ilzll.. for each 3 e 
{1,2.--. ,}. 



2932 SEUNG JUN CHANG, .IAE GIL CHOI, AND DAVID SKOUG 

Next, using (3.8) and (3.9), we see that for p > 0 and h > 0. 
<_ pllzll,»lf(<,,.p., + pi,,,,),..., (o.pa. + ph,v)) i 
Dut this implios that R(px + phwpw), as a fimction of x. is -integrable for ail 
p > 0 and h > O. This tan ho seon by intogrting the right-hand side of (3.10) terre 
by terre. For example, usin (2.20), ve see that for any ! E {1.--- ,}, 

(3.11) 

.exi,{_ - [uJ - P(Aj + h(aJ'w})]}du ...du n 
j= 2p2 B. 
Thus, using (3.10) and (3.11). we ol)tain that for p > 0 and h > 0. 
Next, using (3.8), (2.19). (3.3), and (3A), we see that for all A > 0, 
(3.12) =  [/()  (l. )l() 
+ () (,, )1,()] (: ). 
But, as noted in Remark 3.1 above, for each A  C+, H(A;/) is an element of 
C0(), and so the integrand on the right-hand side of (3.12) is in L(). Hence. 
eF [R(-I,)] = eF [F(')6(I) + F(I)()] 
exists for all   C+. A simil argument shows that the analytic function space 
integral E[F(x)G(x)] also exists for all   C+. Equation (3.7) nowfollows 
from Theorem 2.1 above; in particular, ri'oto equation (2.17) with F(x) replaced 
with R(x). 



INTEGRATION B PARTS FORMULAS ON FUNCTION SPACE 2933 

The following two corollm'ies are special cases of Theorem 3.3. 

Corollary 3.4. Let z, w, and m be as in Theotvm 3.3. Let F  B(2: m) be given 
by (1.1). Then for all A • C+, 
(3.13) AEn x  
= _ [(F(x)) (z,x)]- (:,a « [(F(x))/. 
Proof. In Theorem 3.3, «hoose p : 2 and Gx) 

Corollary 3.5. Let z and z2 bc elem«7,ts of L.b[O.T], and for t • [O.T], let 
wj(t) : fO zj(s)db(s) for j • {1.2}. Let F • (2:,n) be given by equation (1.1). 
Then ,for all A • C, 

(3.14) 

= AEaxn[F(a')SF(xlu,1)(z,x)]- V/--(z,a')En[F(a.)(F(X[Wl)]. 

Pro@ Let I' = 2 and G(x) = F(:rl., ) in Theorem 3.3. 

Lemlna 3.6. Let p. m and F be as in Lemma 3.1 obove. Then for all A • C, 

(3.15) 
for s-a.e. 9 • C,b[O. T] "u,herc 

(3.16) 

,. = f(ff + OH(A: )dg[ 

with t?j and H given by eqaations (2.19) and (3.4) respectively. 

Pro@ For A > 0, equation (3.15) follmvs easilv fronl equation (2.20). But for 
each A • C+, as shown in Remark 3.1 above, H(A; u,--- ,u,) is an element of 
L()aC0(l ) for all p • [1. oe]. Hence, for each A • C+ and s-a.e, y • 

f(//1 -[- (Ctl,/},---,/t n n c (Ctn,//))//():U1, --./An) 
belongs to L(NI ') and so equation (3.15) hohls throughout tE+. 

Our next lemma follows from standard results for convolution products. The 
key is that fol each A • C+, H(A: ff) is an element of L»(]K ') ç C0(]K n) for ail 
1 <_p<_ +oc,. 

Lemma 3.7. Let cho 
(a) If f • L  (]R ). 
(b) If f • LP(IR ) 
where p' = P 
p--l" 
(c) If f • 

be given by equatioz (3.16) above. 
then ¢0(A; ") • C0(]R ') for all A • C+. 
for some p • (1, ac), then b0(A:-) • LP'(IR ") for all A • C+ 

then bo(A;-) • LI( n) for all A • C+. 

Our next theorenl follows immediately frolll Lemnla 3.7. 

Theorem 3.8. Let 1 <_ p <_ 



2934 SEUNG JUN CHAN(], JAE GIL CHOI, AND DAVID SKOUG 

Theorem 3.9. Let 1 < p < oe and w • A be gwen. Let F • B(p; m ) be gzven by 
«quatzo, (1.1). Then for all A • C+ and s-a.e, y • C«b[O,T], 

(3.17) 

, W) fl(al -- /l, y), - - - ,1tri -- 

-H(A;al,'-- ,an)dul...dun 

= T(5f(-I,,))(v). 

which, as a function of g. is an element of B(p'; m- 1). 

Proof. The fact that 5T.x(F)(y]w) is an element of /3(p';m- 1) follows directly 
from Theorem 3.8 and Lemma 3.1. To establish equation (3.17) for A > 0. simply 
ca.lculate 5T(F)(y]w) using equation (3.15), and then calculate T)(5F(.Iw))(y) 
using equations (3.1) and (2.9). Finally. equation (3.17) holds throughout C+ by 
analytic continuation in A. [] 

In our next theorenl we obtain an integration by parts formula involving Tx(F) 
and T,(G). 
Theorem 3.10. Let p. m, z, w, F and G be as in Theorem 3.3. Then for all A • C+, 
(3.18) 
E" [T(F)(x)ST(C)(xlw) + 5T(F)(x]w)T(C)(x)] 
= AEï'[T,(F)(x)T(G)(x)(z,x)]- OE(z,a')E''[T,(F)(x)T,(G)(x)]. 
Proof. For x • Ca,b[O.T], let R(x) = T(F)(x)T(G)(x). Then by Theorem 3.8, 
T(F) • /3(p'; m) and T(G) • /3(p: m). Hence, bv Lemma 3.2, R belongs to 
(1; rn), and so by Lemma 3.1.5R(x[w), as a fimction of x, belongs to (l'm- 1). 
Thus, equation (3.18) follows from Theorem 3.3 with F and G replaced by T,(F) 
and T.,, (G) respectively. [] 

Theorem 3.11. Let m,z and w be as in Lernma 3.2. Let p • [1.2] and let F 
and G in B(p:m) be given by equations (1.1) and (3.2) r«sp«ctively. Then for ail 
A•C+, 
E.'[F(x)A(C)(xlw) + F(xlw)T(C)(x)] 
(3.19) 
= t'[F(.)T()(.)(,.)]- OE( -"-',, 
z,. )r.. [F(x)T.(G)(x)]. 
Proof. Let R(x) = F(x)T(G)(x) for x • Ca.b[O,T]. Bv Theorem 3.8, T(G) is an 
element of/3(/;m) and hence bv Lemma 3.2, R belongs to /3(1; m). Hence, by 
Lemma 3.1, 5R(x]w), as a function of x, belongs to/3(1"m - 1). Thus, equation 
(3.19) follows from Theorem 3.3 with (7 replaced bv T,((7). [] 

Corollary 3.12. Let m, z, w,p and F be as in Theorem 3.11. Then for ail A • C+, 

(3.20) 

E'[F(x)5T(F)(xlw) + 5F(x[w)T.(F)(x)] 
= E'[:()T(F)(.)(z,.)]- OE(z..')F[F(.)T(F)()]. 

Pro@ Simply choose G = F in Theorem 3.11. [] 



INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2935 

Corollary 3.13. 

Let m, z and w be as zn Lemma 3.2. Let F E /3(2;m) be given 

2 

by equation (1.1). Then for all A  C+, 
Eoe n: [TA(F)(x)6T(F)(x]w)] 
(3.21) A 
__-- __ Ean. 
2  [(TA(F)(x))2<z'x] 

Proof. Siinply choose p = 2 and G = F in Theoreln 3.10. 

4. PARTS FORMULAS INVOLVING Tq(l: F) AND Tq(l: G) 
In this section we obtain various integration by parts fornmlas involving the 
analvtic GFFTs Tq(1; F) and Tq(l: G). In view of equation (3.6) above, we clearly 
need to impose additional restrictions on the functionals F and G than were needed 
throughout Section 3. 
Fix q e - {0}. Then as A  -iq through values in C+, « = Re()  /2 
and Idl  /2 where d = hn(). 
Next using equations (3.3) through (3.6) we see that for all A G Ç+ with c = 
Re() < ((1 + 

]H(A;ff) =exp{- 

(4.1) G exp 

<_ exp 

In addition. 

(4.2) 

1 

For f  La(IR ') let 
(4.3) 9r(/)(O = (27r)- 
denote the Fourier transform of f. 
Theorem 4.1. 
(1.1) with 

(4.4) 

, f(ff)exp { i Z ujçj }d 
j=l 

Let q  IR - {0} be given. 

Let F  /3(1; m) be given by equation 

1 



2936 SEUNG JUN CHANG, JAE GIL CHOI, AND DAV|D SIxOUG 

for all k ¢ {0, 1,--. ,m} and each j  {1,--- .o}. Furthermore, assume that 

belongs to Co(lI). Then 
(4.) ¢o(-«:4) --_ _ -- 

f((+ ff)H(-iq: )d6 

is an, element of Co(II). Furthermore, the L1 analgtic GFFT, Tq(1; F) exsts as 
an elemeot of/3(c: m) and for s-a.e, g  Ce,bi0. T] is givet by tbe formula 

(4.7) Tq(I: F)(y) = ¢0(-1q; <al, g),'- • , <cn, g)). 
Proof. By (4.1) and (4.4) we know that f(.)H(-iq: .)  L(]n). and so its Fourier 
transform, ,T(f(.)H(-iq:.))(O cxists and bclongs to C0(]). Furthermore, bv 
cquations (4.6) and (3.4) and thc fact that vfZiq = c+ di = V/2 + di, we obtain 

1 

(4 8) 

- exp 

J= 2Bi B " ' Bn " 

Bv assumption (4.5), it follows that d)0(-iq: 0 is an element of Co(n). 
Finally, by equations (2.11), (3.15), (3.16), (4.8) and the dominated convergence 
theorem (the use of which is justified by (4.2)), it follows that for s-a.e, g  



INTEG1RATION B PARTS FORMULAS ON FUNCTION SPACE 2937 

Theorem 4.2. 
more. assume that for each 1 • { 1, 2,-.- , n }, 

Let q • IR - {0} and F • B(I m) be as m Theorem 4.1. Further- 

{ } ( ) 
q'l q n 
(4.10) exp - 9-[qAjJ (fl(.)H(--iq;.))- , ..... 

belongs to Co(IRn). Tken for eack 1 • {1,2,---,n}. 

(4.11) çt(-iq:() = _  , ff((+ ff)H(-iq:ff)di[ 
is an element of Co(Rn). In addition, for each w • A and s-a.e. 
5Tq(I: F)(y]w) =  (t, w}Ot(-iq: (a,y), . - - , (a.,y)) 
(4.12) =l 
= (. aF(.I))(v), 
which, as a fuc*io of g. is a element of B(;m- 1). 

Pro@ The proof that each çbt(-iq; .) belongs to Co(IR ') is the saine as the proof in 
Theorem 4.1 above showing t hat ¢0 (-iq:') • Co (IR '). Equa.t ion (4.12) t hen follows 
immediately using the defilfition of the first variation and equa, tion (4.7). [] 

Our next theorem gives a parts formula involving F and Tq(I: G). 
Theorem 4.3. Let q •  - {0} be given and let F • B(I: m) be as in Theorem 

4.1. Let G • B(I: m) be given by equation (3.2) with 

(4.13) 
for all k • {0, 1,--- ,m} 
(4.14) exp { - -Ç 

1 
2 =1 

and each ji • {1,--. ,ï}. Furthermore, assume that 

2X/-[Ajj } -(gl(.)H(_iq: .)) ( _ ql 



2938 SEUNG JUN CHANG, JAE GIL CHOI AND DAVID SKOUG 

belongs to Co(N n) for all l 
t e [O,T], let w(t)= f z(s)db(s). Then 

Lêt z  L,[O.T] be gwen and for 

E'fq[F(x)STq(1; G)(xlw ) + 5F(x[w)Tq(1; G)(x)] 
(4.15) = -iqEnfq[F(x)Tq(l: G)(x)(z,x)] 
- (-iq)(z.a')E'f[F(x)Tq(l'G)(x)]. 
Pro@ Let R(x) = F(x)Tq(l'G)(x). By Theorem 4.1. Tq(l:G)(x) is an element of 
B(; m) and so R(x) is an element of B(I: m). Also, by Theorem 4.1, Theorem 4.2 
and Lemma 3.2, 
6(x,,,) = F()6T(I: (1,) + 6F(,)T(: a)(x). 
as a fimction of x, is an element of B(I" m - 1). In addition, we know that for each 
I e {0.1,.-. ,.}, 
1/2 
Je[- 9(ff+ ff)H(-iq:ff)dff 

/'t(-iq:ç = _ 27rBj 
is an element of C0(IR n) with 

and 

for s-a.e. 
exist: 

(4.16) 

and 

Tq(l:G)(y) = bo(-iq: (c,y),-.-, 

6Tq( l: G)(y]w) = Z (ct, w)g't(-iq; (c,, y), . . . , (c,, y) ) 
/=1 
y E C,,b[O. T]. Hence, both of the following analytic Feynman integrals 

Exanfq[/(X)] = Enfq[F(x)Tq(l'a)(x)] 

= ( I @ ) /2 ,, f(ff)bo(-iq: ff)H(-iq', ff)dff 
j=l 

Exanfq [(!(X1732)] = Enf[F(x)STq(1; G)(xlw ) + F(xlw)Tq(l: G)(x)] 
- -(,-iq)/£[ 
(4.17) 
-- 
+ 0(-iq:ff) (at,w)ft(ff) H(-iq:ff)dff. 
/=I 
Also, proceeding as in the proof of Theorem 3.3 above, it is easy to show that for 
p>0andh>0, 
E[[6(px + pbwpw)] 
(4.18) _ 
+ 
/:1 



INTEGRATION B PARTS FORMULAS ON FUNCTION SPACE 2939 

Hence, by Theorem 2.1 above, the analytic Fe.wmmn integral 
E'G[R(x)(z,x)] = 
exists and equality (4.14) holds. [] 
Choosing G = F in Theorem 4.3 we get Ihe following integration by parts for- 
lnula. 
Corollary 4.4. Let q C OE- {O} be given and let F  B(1;m) be as i Theorem 
4. `) . Let z ad w be as in Theorem 4.3. Thez 
Eaznfq [F(:r)STq( l : F)(x[w) +  F(x[w)Tq(1; F)(.r)] 
(4.19) = -iqE"f[F(.r)Tq(l: F)(x)(z,x)] 
(-iq)½(z, ,,a,f 
- a lr, [F(.r)Tq(l:F)(z)]. 
Next we obtain a parts formula involving Tq(l F) and Tq(l: G). 
Theorem 4.5. Let q  OE- {0}. Let F  B(I: m) be as in Theom 4.2 and let G  
B(I: m) be as in Theorero 4.3. Furhermore, assume Hmt .for each l  {0.1,---  n}, 

(4.20) 1, .@(-iq: ff)H(-iq; ff) dg 
Then for w(t)= fô z(s)db(s) with z 

(4.2) 

Pro@ Let R(x) = Tq(l:F)(z)Tq(l:G)(x). Then R G B(oc,: m) and 6R(.rlw ), as a 
function of :r, is an element of B(oc;m- 1). Hence, by (4.6), (4.11) and (4.20), 
both of the following analytic Feymnan integrals exist: 

(4.22) 

and 

(4.23) 

1 
j=l 

b0(-iq: ff)ç'0(-iq; ff)H(-iq: ff)dg 

_ 2rrBj 

i 13. 
 qS°(-iq: g) E(at" w)@(-iq', g) 
n 1=1 

 ] 
+ bo(-iq; ff) E(oq, w)@(-iq;) H(-iq:)dff. 
/=1 

In addition, for p > 0 and h > 0, 
< pllzllo.» [],;bo(-iq; )ll,  II@(-iq; )11, 
(4.24) /=1 
+ II'b0(-/q; ")lloe E Ilçl(-iq; 



2940 SEUNG JUN CHANG, JAE GIL CHOI, AND DAVID SKOUG 

Hence, bv Theorem 2.1, the analytic Feymnan integral E  [R(x)(z, )] exists and 
equality (4.21) holds. 
 finish this section with some examples which shed light upon the necessity 
of conditions such  (4.4) and (4.5), and which also illustrate that the conclusions 
of Lemma 3.7 are hot necessarily valid for A Ç Ç+ with e() = 0. 
In our first exainple ve define a flmctional F of the form (1.1) with n = 1, such 
that F is an elelnent of B(p; m) for ail p Ç [1, +], f is an element of L»(N) for 
all p  [1, +], and yet o(i; ") given by (4.6) is hot an element of Co(). In fact, 
[o(i:)1 = +OE for all ( Ç N. 
Exalnple 4.6. Let q = -1, let 't = 1, let mbe a nonnegative integer, and let 
be an elelnent of La»[0, T] with IIllla» = 1. Without loss of generality, we will 
smne that ,4 (see equation (2.18)) is positive. 
Let f " N + C be defined bv the fornmla 
{iuî iA,u, A AlUl } 
(4.25) I(,)  ç+Xo,+)()p 2e 2 + 2 4 " 
 note that 
2B1 4B " 
and hence f Ç LP(N) for all p Ç [1, +oe]. In fact, f is also an element of Co(N). 
ç then define F • C,[0. T]  C by the formula 
(.) F()  I((,, )). 
It is easv to see that F Ç B(p: m) for all p Ç [1, +]. 
Next, using equation (3.4) with  = 1, A = i, and  _ +i, we observe that 
(4.28) H(i; ul) = exp { Aul + iAu - A - iuî } 
2B " 
and hence 
(4.29) f(Ul)H(i:l)=U+'[o,+)(l)exp{ A } 
4B1 " 
which is not an element of Lp(N) for anv p 
Then, using equation (4.6) with n = 1 and q = -1, equation (4.25) and equation 
(4.28), we s that 
1 
() 
(i;) = 2B f(u +)H(i;ul)du 
1 
= 2ÇB exp  2B 
(4.30) - (u +)+k[o,+)(u +)exp iu + du 
B 4B 
1 
() {«î « 
=  exp 
2B 2B 
- (.  )+ ex  + . 
ç B 4B1 



INTEG1RATION BY PARTS FORMULAS ON FUNCTION SPACE 2941 

Thlls, 

(4.31) 
/_t-(x {tLtll v/Alttl } ] 
(Lt 1 ÷ 1) m+l exp -- + du.1 . 
• 1 B1 -lB1 
Hcnce, choosing 1 - 0, and using the fact that Al is positive, we see that 
lao(i: O)l = (2B,)- u m+x exp 
: (2gB1) - 0 m+l exp  du = +, 
which implies that 0(i; ") is not an element of C0(R). hl fa«t. for each fixed   R, 
w observe that 
,0(/;,),=(2B)_exp { .4,1}4B1 
(4.32) 
(Ul+ exp + dul = 
and so 00(i; ") is hot an elelnent of LP(N) for any p  [1, +] even though f(.) was 
an elelnont of LP(N) for all p  [1. +] and F was an elclnent of B(p: m) for all 
p  [1, +oe]. Hence, the L1 analytic GFFT, T_(I: F) does hot exist. 
ç also note that f does hot satis6" condition (4.4) above since by equation 
(4.26) (recall that q =-1 and so ()/2 = 1), 

 lf(ul)]exp { IAlUll } dulB 

"tlUl } 
v"2.4ul + __ dal = +oc. 
4B1 BI 

In our lmXt example we exhibit a fUlctional F of the forln (1.1) that satisfies 
conditions (4.4) and (4.5) above. Furtherlnore, we are able to evaluate the integral 
in equation (4.6) and thus obtain a forlnula for ¢0(i: ) which does not involve anv 
integrals. 

Example 4.7. Let q = -1, let m be a nonImgative integer and let r be a posi- 
tive integer. Let {ai,---, a} be an orthonolnml set of functions froln (L.b[0. T], 
1" [«,b). and for ea.ch j e {1.---.n} let Ai and B be given by (2.18) a.nd (2.19) 
respectively.  define f : N   Cbv the forlnula 
[u - iAuj + A - u -- 
(4.33) f()  exp . 
J= 2B 

}Ve note that 
(4.34) 

j=l 2Bj 

and hellCe f e LP(] n) for ail p e [1, +o1. Also, f e Co(IR). 



2942 SEUNG JUN CHANG, JAE GIL CHOI, AND DAVID SKOUG 

Let F " C,.b[O, T] --, C be given by 

(4.35) 

F(X)  f((l,x),"" ,(a.,x)). 

It is easy to show that F • B(p; m) for all p • [1. 
Next, using equation (4.33), together with equation (3.4) with A = i and x/ï = 
1+{ it follows that 

(4.36) 

f(ff)H(i; ff) = exp { - uj 

Now clearly f(.)H(i: .) is an element of LP(N ") 0 Co(N ") for ail p • [1. +oe]. Next, 
using equations (4.6), (3.4) and (4.33) we obtain 

(4.37) 

• f exp i Ujj 
1 j:l 

because 

(4.38) 

HClIC% 

(4.39) 

Ioo(i;()1 =exp- 2Bi ' 
j=l 

and so 00(i; ") is an element of C0(lI n) N LP(I n) for all p e [1, +oc]. 
2 
We also note that because of the factor exp{ 2--ï, } in the definition of f(ff) 
given by equation (4.33), condition (4.4) certainly holds. In addition, condition 



INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2943 

= Bj 1¢0(i; 

Hence, by Theorem 4.1. the Lt 
g • 

5. PARTS FORMULAS INVOLVING Tq(2;/) AND T(2:G) 

Note that in OllI" first theorem below we replace conditions (4.4) and (4.5) with 
condition (5.1). This condition is used to obtain a dominating flmction in order to 
apply the dominated convergence theorem. 

Theorem 5.1. Let q • IR - {0} be given. 
(1.1) with 

Let F • B(2: m) be given by equation 

for all k • {O. 1,... ,,n} and each j • {1.---,,z}. Then 
1 
(5.2) ¢o(-iq;& = _ 2rrB) ,, f((+ ff)H(-iq: g)dff 
is an element of L2(N). Furthermore. the L2 analytic GFFT. Tq(2: F) ests as 
an element of B(2ç m) and for s-a.e, y  C,b[O, T] is given by the formula 

(5.3) Tq2: F)(y) = ¢o(-iq; (c, y),. -- , (a,, y>). 
Proof. Using (4.1) we first note tllat 

If((+ ff)H(-iq: ff)ldff 

f((+ff)l exp {(1+}q1)2 

J= Bj 

1 
léo(-iq()l-< _ 2-7-D-7) 
1 
j= 2roB9 ] 

dff. 



2944 SEUNCI JUN CHAN(3, JAN GIL CHOI, AND DAVID StxOUG 

Hence. by (5.1) with k = 0. 

iàlld so 00(--iq;() is ill elen,t'nt of L2(lIr). 
To show that Tq(2: F) exists and is given by equation (5.3) it stlflîces to show 
that for ea«h p > 0. 

}ira /[ TX(pq) -cO(-iq; (o,.p./),'" , ((rt. py))12dl(l) 
-\--q. »[0,T] 

But 

No,v clearlv 4,0(A:)  00(-iq:) a.e. o,, IR  as A -- -iq through vah, es in C l . 
Tlms, to show that 

114,o(,x; .) - 4,o(-iq;-)112 ---' o, 

it suffices [11, p. 126] to show that 

114,0(,x: )112 --+ 114,0(-iq: )112 

asA -.iq through vah,es in C+. But forall A G C+ with Re(v/ï) < ((l+lql)/_)-. 

"9 
j=l 



INTEGRATION BY P,RTS FORMULAS ON FUNCTION SPACE 2945 

Hence, b¥ the dominated convergence theorem, 
lira 

0H(A: 

2 

5")dff 2 
f(ff + OIq(-iq; 

Corollary 5.2. Let q E IR - {0} and F  /3(2; m) be as in Theorem 5.1. Then for 
each !  { 1,2,..- , n}. 
1 
j=l 
is an element of Lu(N'). In addition, for ea«h w e A and s-a.e, g e Ce,bi0. T], 

(5.5) 

5Tq(2; F)(ylw ) = Z(a' w)¢,(-iq; (a,,g),-.- , 

= Tq(2:5F(.Iw))(y ), 

which, as a function of g, is an eleme'nt of/(2: m- 1). 

,)) 

Pro@ The proof that each ¢(-iq;-) belongs to L2(IR n) is the saine as the proof 
in Theorem 5.1 above showing that ¢0(-iq-')  Lu(IR'). Equation (5.5) then 
follows inmmdiately using the definition of the first variation and equations (5.2) 
and (5.3). [] 

Theorem 5.3. 
5.1. Furthermore, assume that 
(5.6)  [f ),,...,j (ff)H(-iq: ff)[dff < oo 
for all k e {0,1,---,nz} and «ach ji e {1,2,--. ,n}. 
Le G  8(2; m) be 9iven b equation (3.2) with 
1 
(5.7, £, [£ }gj,.....j«((+ff, lexp{ (1 ,q,)  
»r all k e {0,1,--. ,m} and «ach ji e {1,---,n}. 
4.3. Then 

(5.s) 

Let q  IR - {0} be given and let F  /3(2:m) be as in Theorem 

IAufil d d( < oc 

Let z and w be as in Theorem 



2946 SEUNG JUN CHANG, JAE GIL CHOI, AND DAVID SKOUG 

Proof. By Theorem 5.1, for each I E {0, 1,--. ,n}, 
1 
(5.9) g,,(-iq;()= (fi)  
j=l 

t, gt(( + ff)H(-iq; ff)dff 

alld 

Exanfq [/(X)] = 27rBj 

Also, for p > 0 and h > 0, 
<_ pl[z[I«»( 1-I 2P2B) -½ IIf[12  ]]t(-iq;-)]12 
3=1 /=1 
+ ][o(-iq; ")112  liftIl2 
l-1 

/(ff)H (-iq: ff)Z/'0 (-iq: ff)da 

£ f(ff) 

+ "0o(-iq; g)Z(at, w)ft(ff) H(-iq:)d. 
l=1 

Hence, by Theorem 2.1, the analvtic Fevmnan integral E'q[R(x)(z,x)] exists and 
equality (5.8) holds. [] 
Next, choosing G = F in Theorem 5.3, we obtain the following integration by 
parts formula. 
Corollary 5.4. Let q. F  B(2: m), z, and w be as in Theorem 5.3. Then 
Enf[F(z)6Tq(2; F)(zlw) + 6F(zlw)Tq(2; F)(z)] 
= -qEnf[F(z)Tq(2; F)(x)(z,z)] 
- (-iq) (», ')z2e [F()T(: 

is ai1 elelnent of L 2 (11 ). Furt hernlore, 
(5.10) Tq(2: G)(x) = b0(-iq; (al.x),'." , (,x) ) 
is an clement of B(2; m), raid as a flmction of x, 
(5.11) 
/-----1 
belOllgS to (2;m- 1). Hel,Ce, /(«)= F(«)Tq(2:G)(x)is al, elenmnt of B(l:m) 
alld 
tiR(xlw) = F(z)6Tq(2: G)(.rlw) + 6F(xlw)Tq(2: G)(x) 
is ai1 eh?lllPnt of B(1; m,- 1). Since 
f(ff)H(-iq; ff)bt(-iq:ff) and ft(ff)H(-iq: ff)b0(-iq: 
belong to L(1R ) for eaçh / Ç {0, 1,-.- , u}, both of the following analvtic Fevmnan 
illtegrals exist: 



INTEGRATION BY PARTS FORMULAS ON FUNCTION SPACE 2947 

Our final theorem is a counterpart to Theorem 4.5 above. 
Theorem 5.5. Let q C ]-{0} and let F and G be as in Theorem 5.3. Furthermore, 
assume that for each l  {0, 1,--. ,n}, 
./ ]¢,(g)(-q: )]d < . 
(5.12) 
Let z G L,b[O,T] be gwen and fort e [O,T] let w(t)= fô z(s)db(s). Then 
Efq[Tq(2: F)(.r)STq(2: G) (x[w) + 5Tq(2; F)(.rlW)Tq(2: G) (x)] 
-qE [Tq(2; F)(x)%(2:G)(x)(z,x)] 
(5.13) = - anf 
(--iq)  (Z, t,--anfq 
- « )x [Tq(2: F)(x)Tq(2:G)(x)]. 
ProoI. Let R(x) = Tq(2:F)(x)Tq(2:G)(x). Then R G B(1;m) and 6R(-[w) G 
E " (x 
B(1;m- 1). Using (5.2), (5.4), (5.9) and (5.12), we sec that [ )] and 
E"[6R(x]w)] both exist and are given bv equations (4.22) and (4,23) rcspec- 
tivclv. Finally, we sec that (5.13) follows from Thcorem 2.1, since for p > 0 and 

h>O, 

We finish t his paper wit h some very brief couunents about the functionals defiued 
in Examples 4.6 and 4.7 for the case p = 2. 
We first note that for the functional F(x)  B(2; m) defined by equation (4.27) 
with f(u)  L2(N) given by (4.25), the L2 analvtic GFFT, T_(2;F) does not 
exist because I¢0(i;)1 = +oe for each   N. In fact, the Lp analytic GFFT, 
T_(p; F) does not exist for anv p  [1, 2]. 
On the other hand, it is quite easy to see that condition (5.1) holds for the 
fun«tion f(ff) given by equation (4.33). Hen«e, for F(x) defined by equation (4.35), 
the L2 analytic GFFT, T_(2:F) exists as an element of B(2;m) and for s-a.e. 
y  C«,b[O,T] is given by the right-hand side of equation (4.41). In fact, for all 
p C [1,2], the Lp analytic GFFT, T_(p: F) exists as an element of B(p';m) and is 
given by the right-hand side of equation (4.41). 

EFERENCES 

[1] R.H. Cameron and D. A. Storvick, An L2 analytic Fourier-Feynman transforrn, Michigan 
Math. J. 23 (1976), 1-30. MR 53:8371 
[2] __, Feynznan integral of variations of functions, in Gaussian random fields, Ser. Prob. 
Statist. 1 (1991), 144-157. MR 93b:28035 
[3] K. S. Chang, B. S. Kim, and I. Yoo, Fourier-Feynznan transform, convolution and first 
variation of functionals on abstract Wiener space, Integral transforms and Special Functions 
10 (2000), 179-200. MR 2001m:28023 
[4] S. J. Chang and D. L. Skoug, The effect of drift on the Fourier-Feynznan transform, the 
convolution product and the first variation, Panamerican Math. J. 10 (2000), 25-38. 



2948 SEUNG JUN CHANG, JAE GIL CHOI. AND DAVID StçOUG 

[5] , Generalized Fourier-Feynman transforms and a first variation on ]unction space, to 
appear in Integral Transforms and Special Functions. 
[6] T. Huffman. C Park and D. Skoug, Analytic Fourier-Feynman transforms and convolution, 
Trans. Amer. Math. Soc. 347 (1995), 661-673. Il=( 95d:28017 
[7] , Generalized transforms and convolutions, Internat. J. Math. and Math. Sci. 20 
(1997), 19-32. M1R 97k:46047 
[8] G. W. Johnson and D. L. Skoug, An Lp Analytic Fourier-Feynman transform, blichigan 
Math. J. 26 (1979), 103-127. MR 81a:46050 
[9] __, Scale-invariant measurability in Wiener space, Pacific J. Math 83 (1979), 157-176. 
MR 81b:28016 
Il0] E. Nelson, Dynamical theories of Brownian motion (2nd edition), Math. Notes, Princeton 
University Press, Princeton (1967). MR 35:5001 
[11] C. Park, and D. Skoug, Integratzon by parts formulas involving analytic Feynman integrals, 
Panamerican Math. J. 8 (1998), 1-11. MR 99i:46031 
[12] H. L. Royden, Real Analyszs (Third edition), hlacmillan (1988). hIR 90g:00004 
[13] .1. Yeh, Stochastic Processes and the Wiener Integral, Marcel Dekker, Inc., New York (1973). 
MR 57:14166 

DEPARTMENT OF IkIATHEMATICS, DANKOOK UNIVERSITY. CHEONAN 330-714. KOnEA 
E-mail address: sejchangdankook, ac. kr 
DEPARTMENT OF IATHEMATICS, DANIxOOK UNIVERSITY, CHEONAN 330-714. KOREA 
E-mail address: jgchoidankook, ac. kr 
DEPARTMENT OF IIATHEMATICS AND STATISTICS. UNIVERSIT5 OF NEBRASKA. 
NEBRASKA. 68588-0323 
E-mail address: dskougmath.unl, edu 

LINCOLN. 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
çolume 355, Number 7, Pages 2949-29ïl 
S 0002-9947(03)03269-0 
Article electronically published on lIarch 19. 2003 

THERMODYNAMIC FORMALISM 
FOR COUNTABLE TO ONE MARKOV SYSTEMS 

MI('HIKO Yi:EI 

ABSTRACT. For countable to one transitive Markov systems we establish ther- 
modynamic formalism for non-H61der potentials in nonhyperbolic situations. 
We present a new method for the construction of conformal measures that 
satisfy the weak Gibbs property for potentials of weak bounded variation and 
show the existence of equilibrium states equivalent to the weak Gibbs measures. 
\'e see that certain periodic orbits cause a phase transition, non-Gibbsianness 
and force the decoEv of correlations to be slow. We apply our results to higher- 
dimensional maps with indiffcrenl periodic points. 

§0. INTRODUCTION 
Thermodvnainic for|nalism for hyperbolic systems was satisfactorily established 
with Bowen's program ([2]). The existence of geuerating finite l\Iarkov partitions 
and analysis of Ruelle-Perron-Frobenius operators associated to H61der potentials 
allow one to show the existence of torique equilibrimn states that satisfv the Gibbs 
property (in the sense of Bowen) and exponential deca,v of correlations. Also, the 
pressure fllnctions are analytic and there is no possibility of phase transition (non- 
uniqueness of equilibrium stat.es). Furthermore, the analyticity pïoblem is strongly 
relateà to nnlltifractal problems and the zero of Bowen's equation determines the 
Hausdorff dimension of linfit sers arising from certain iterated flmctional systems 
([4], [8], [12], [22]). On the other hand, phase transition, failure of the Gibbs prop- 
erty and slow decav of correlations can be observed for many COlnplex svstems 
which exhibit common phenomena in transition to turbulence (the so-called Iz- 
terrn{tte«y). In this paper we shall construct mathelnatical models which exhibit 
such phenomena and for this purpose we shall establish thermodvnanfic formal- 
ism for non-H61der potentials in nonhyperbolic situatious in the following sense: 
generating l\Iarkov partitions are countable partitions and dvnamical iustabilitv is 
sbecponential (subexponential decav of cylinder sizes). More specifically, for count- 
able to one transitive Markov systelns we shall construct conformal measures oe that 
are weak Gibbs Ineasures for potentials q of weak bonded variation (WBV)(see clef- 
initions in §1) and show the existence of equilibriuln states t/ for q equivalent to 
the weak Gibbs measures oe. The conformal lneasures oe associated to q play im- 
portant roles as reference measures from the physical point of view, and absolute 
continuitv of equilibrimn srates allovs one to describe statistical properties of ob- 
servable pheltoInena in the physical sense. In order to clarifv typical reasons for 

Received by the editors Match 28, 2002 and, in revised form, Septlember 10, 2002. 
9 
2000 Mathematics Subject Classification, Primary 8D99, 28D20, 58C40, 58E30, 37A40, 
37A30.37C30. 37D35, 37F10, 37A45. 

(2003 American I\Iathematical Society 



2950 M. YURI 

plmse transition, non-Gibbsianness and slow decay of correlations, we introduce in 
§4 a notion of idifferet periodic point associated to potentials ¢ of VBV. Those 
periodic points cause failure of sunnnable variation for potentials ç and failure of 
bounded distortion of local 3acobians with respect to the weak Gibbs measure u for 
¢ (Proposition 4). Then a construction of weak Gibbs measures u for ¢ admitting 
indifferent periodic points implies subexponentlal mstablht in terres of cylin- 
der mesures (Proposition 3). Furthermore, a construction of equilibriunl states  
for such  equivalent to oe (Theorem 6, Lenmla 15) allows us to show both phase 
transition (Çorollary 2 Theorem 8) and non-Gibbsianless of equilibrimn states 
(Theorem 5). bi particular, our results are applicable to the following piecewise 
CLsmooth countable to one Markov lllap8 T defined on bomded regions X C u 
with indifferent periodic points (T%'0 = x0, det DT(.r0)l = 1) for which the p 
tentials - log  det DT] satisfv neither sunnnable variation nor bounded distortion 
so that previous results cannot be applicable. 
Exau,ple A. (Inhomoge'neous Diopha,tine approximations [13], [15], [16], [17], [19]. 
[20], [21]). Let X = {(x,.q)  N" 0  g  1,-g  .r < -g+l} and define 
T'XXby 
1 - , . 
-- +  ___ 
ZN). This map admiIs indifferent periodic points (1.0) and (-1.1) with period 
2, i.e., IdetDY(1,0)l = IdetDY(-1.1)l = 1 and is related to a Diophantine 
approxilnation problem of inhomogeneous linear class.  
complex continued fraction transforlnation T • ¥  X on the dimnond-shaped 
region X = {z = .ra +" - 1/2 G a.a- G 1/2}, where a = 1 +{, by 
T(z) = 1/z -[1/Zll. Here [z] denotes la, + 1/2]a + [2 + 1/21, where z is 
written in the form z = aa + ,[] = max{n  Z I  G x}(«  N) and 
[] = m{n  Z I n < z}(a  Z- N). This transformation bas an indifferent 
periodic orbit {1,-1} of period 2 and two indifferent fixed points at { and -i.  
ç recall previous works related to thennodynanfic fonnaIism for countable to 
one Markov systems. For countable Markov shifl.s, O. Sarig proved the existence of 
conforlnal measures and equilibrium states associated to locally H61der potentials 
defined in [10] and D. Fiebig, U. Fiebig and the author proved the existence of 
equilibriuln states for potentials satisfying bounded distortion (supnk Çn < OE in 
the definition of WBV) in [7]. Our main Theorems 4-8 do ilOt satis" these as- 
sunlptiollS and Examples A, B show that they cannot be treated by methods in 
[0] and in [7]. rthernlore, for higher-dimelSiolml sy8teillS that are not symbolic 
systems we mav bave crucial difficulties in verifying the positive recurrence con- 
dition impod on potentials in both [10] and [7]. The infinite iterated functional 
systems that Manldin and Urbanski studied in [8] correspond to the local inverses 
of piecewise confornml countable Bernoulli systems in out sense, and the method 
nsed in [8] severely relies on the Bernoulli property which fails to hold for Example 
B. Moreover, a H61der-type condition, imposed on potentials for the existence of 
con%rmal measures and %r establishing a variational principle, is ilOt satisfied bv 
the important potentials -log ] det DTI for both Examples A and B. 



COUNTABLE TO ONE MARKOV SYSTEMS 2951 

In order to prove our theorems, we first give iii §2 ail appropriate definition 
of topological pressure for countalle to one transitive Markov systems with finite 
range structure. Our definition coincides with the standm'd one by using peri- 
odic points under certaiu conditions (Lemma 6) which tan be easily verified for 
higher-dimensional exanlples in §8. We also associate the topological pressure to 
the spectral radius of the Perron-Frobenius-lRuelle operator (Theorem 3). The es- 
sential issue for constructing both weak Gibbs measures u for 05 and equilibrium 
states p. for 05 eqnivalent to u is to derive Schweiger's jump transformations T* over 
full cylinders (see the definition in §1) with respect to which a local exponential 
instability (05) and a local bou'nded distortio (06) for potentials ¢ are satisfied. 
Then showiug the existence of a zero of a generalized Bowen's equation (GBE) for 
derived potentials ç* associated to T* (Lemnm 7) allovs one to show the existence 
of conformal measures that are weak Gibbs measures for  (Theorem 4). Under a 
mild condition which cannot be covered bv previous works, we show the existence 
of a zero of (GBE) in §3 by using a product formula of zeta funct.ions (Proposition 
1), which shows a nice relation between zeta functions for the original systems and 
zeta functions for the jump transformalions. We also construct a-finite conformal 
measures via induced maps TA over a single full cvlinder .4 (Theorem 7) in §6 by 
using some idea that appeared in a previous work bv M. Denker and the author 
[5] in which no evidence of the existence of weak Gibbs confornml measures was 
given. We establish the existence of equilibrium states p for 05 of WBV equivalent 
to the weak Gibbs measure u for 05 via a junlp transformation (Theorem 6) in §5 
and via induced maps (Lemma 15) in §6. Theu we can immediately see that the 
appearance of indifferent periodic orbits associated to 05 implies a phase transi- 
tion, i.e., failure of the uniqueness of equilibrimn states (Corollary 2. Theorem 8). 
We should remark that our construction via induced maps shows the existence of 
(countably man3- ) lnutually singular equilibrum states. In §8 we apply our results 
to higher-dimensiolml piecewise C  Markov maps with indifferent periodic points. 
AI1 proofs of results in §§2-3 are postponed to the Appendix. 

§1. PRELIMINARIES 
Let. (X, d) be a compact metric space and let T • X -+ X be a noninvertible map 
that is hot necessarily continuous. Suppose that there exists a countable disjoint 
partition Q = {-¥i}iÇI of ,¥ such that iÇI intX is dense in X and the following 
properties are satisfied. 
(01) For each i  I with intXi ¢ O,T[itx, "inlXi  T(intXi) is a holneomor- 
phisln and (T]x,)- extends to a homeolnorphisln ri on cl(T(inlX)). 
(02) 
(03) {Xi}ieI generates , the a-algebra, of Borel subsets of X. 
We say that the triple (T, X. Q = {X}eI) is a piecewise C°-invertible system. 
By (01), F[tx, extends to a holneomorphisln (v) - on cl(intXi) for i e I 
with intXi ¢ O. For notational convenience we denote (v) - = Tcl(zntx). Let 
i = (i ... i)  I' satisfy int(X  T-1Xi ... T-(n-)Xi) ¢ O. Then we define 
X := Xii  T-1Xi=  ...  T-(n-1)Xi, which is called a cylinder of tank n and 
write Iii = - UN (01), TI,,x,1 ..... • imX,1...i  + ç(int(X...,)) is a homeo- 
morphism and (Tn]intXil...i)-I extends to a homeonorphism Vil o vi= o ... o vi = 
Vil...i,"cl(T(int-k))  cl(intXt) and (Vil...in) -1 = rl,,,x...,.» x> impose 



2952 M. YU1RI 

on (T, X, Q) the next condition, which gives a nice comltable states swnbolic dy- 
nallli£s Sil,lilar to SOfi£ shifts (cf. [15], [161, [171, [19]): 
(Finite Range Structuv).  = {it(T"Xh....) " VXh....,Vn > 0} consists of 
finitely many open sul)sets U1, .... UN of X.  
In particular, if (T, X. Q) satisfies the Markov property (i.e., inXi  intTAÇ.  
0 ilnplies inTXj D idX), then lg = {it(TX) • Vi  I} and we say that (T, X. Q) 
is an FRS Markov sys*«w.. If Ni  Q satisfies cl(T(i*Xi)) = X, then Xi is called 
a ,[vll clinder. If all cylinders arê full cvlinders so that  = {iLY}, then (T, X. Q) 
is called a Beruoulli sWtem. " assulne further the next transitive condition: 
N 
(Transitivitg). btX = U= ut and Vl E { 1,2 .... , N}, 30 < s < OE such that for 
each k E {1,2,..., OE'}, U contains an inlerior of a cylinder ,Y(t0(s) of rank s 
such that T ' (itX(k')(sz)) = U.  
The transitivity condition allows one to establish the next fact. 
Lenllna 1. Therc exists (} < 5' <  su«h that Ts(Uï= intX(k'O(s)) = intX and 
VXi  Q. TS+Z(idX,.) = intX. 
Pro4 4 L«mma 1. Since each Uk contains Uï=] intX(k'l)(,), choosing S =  st 
is enough t.o establish the desired fact.  
Re'mark (A). If (T, X. Q) is a Markov systeln, lhen lhe transitivity condition implies 
«periodicitg in the following sense: S > 0 such that VU, U  H,V > S.  = 
(iZ ...in) with itX ¢  satis{ving btXi, C Uk and T(int(Xi,)) = U. 
Definition. } say that  is a potential of weak bouuded variation WBV) if there 
exists a sequence of posit ive lmmbers { Cn } sat isfying lim  (1/ ) log C = 0 and 

Vr k 1,VXi,...i E V:o T-]Q, 
StlPz6xi .... 
illf«6xq .... 
Define 

temarl (t). 

n--i 
exp(Ej=0 ¢(T  x)) 

n-1 

If lar(T.b) - 0 as n -- ec,, which implies COlltilmity of ¢ iii 

symbolic distance, then çb satisfies the \VBV property. Hence if (T, X.Q) is a 
subshift of filfite type, then anv contilmous ftlllCtiOllS satis[v the WBV property and 
if (T. X. Q) is a countable Markov shift, then any unifornflv continuous functions 
 with l'oq(T, ¢) <  satisfv the WBV property ([7]). 
Let  be the «-algebra of Bord sers of the COlnpact space X. 
Definition ([17], [18], [20]). A proba.bility lneasure p on (X,) is called a weak 
Gibbs measure for a fllnction ¢ with a constant P if there exists a sequence {K }>0 
of positive numbers with lim+(1/n)logK = 0 mlch tha.t -a.e.,r, 
ç2' < (X'" 0"))  çn, 
where Xi...i. (a') denotes the cylinder containing m. 

l'a,'n(T, 0) := sup sup le(x) - 0(g)l- 
}--- ,n--1 
eVj=o T-J(Q) ac,yEY 



COUNTABLE TO ONE MARKOV SYSTEMS 2953 

For a fimçtiou b • X - OE, we define al opertor £ iy 
If  satisfies l'ar(¢) + 0 (, + ), II£lll := sup.x £4l(.r) <  and 
(04) {t'i}iG I i tll equi-cmlimous familv of partially defilwd mfiformlv «onlilut- 
O/lS 
then £ç preserves C(X) (i.e., £ " C(X) + C(X)) and is callcd thc Ruelle- 
Perrot-Frobenius operator. Ve rcmark that (04) is valid if 
V=o T-2(Q)} + 0 as . + 
 recall the uexl result, whi«h follows rioto Theorem 5. l in [17] and Propositiou 
. i [18]. 
Len,ma 2 ([17], [18]). Let (T.X.ç) be a t,'a,,.sitiee FRS Ma,'t'oe 
intX G . and let  be a potential qf IVBI'. Assume tlmt lh«re e.rist p > 0 and a 
Borel pmbabilit9 mea.sure u ot (X. U) sati.sf9inq £u = pp. whcre £* is the dual of 
. 
£. Thetz œe i.s a weak Gibbs mca.ure .for  with -logp. 
Definition.  say lhat a Borel probalfility mcasure u on X is at f-conformal 
measuzv if d(uT)] , 
In order to show the weak Gibhs prol)erty of oe. we use thc [ollowing formula of 
thc local .]acobians with respect t() oe • 
dx, 
Thus for the existence of weak Cil»bs measurcs, it is enough to show the existence 
of conformal measures (see 3). 
Lemma 3 (Theorem 2.2 iu [18]). Let u be a 'w«ak Gibbs measure for O u,«th -P. 
If there e:eist.s a T-ittvariant ergodic pmbabilit9meastte i t equivalent fo u with 
In particular, if the constant Pis the measme-theorctical pressure, thon the 
existence of a T-invariant ergodi« probability measure i t equivalcut to thc weak 
Gibbs measure u for ¢ with -P implies thc existence of a« cquilibrium state for 
ç (see 4). Iu order to achievc both coustructions of conformal measures and 
equilibrium states, we need to introduce new dcrivcd systems which are callcd jtmp 
transformatios ([13]). Let B C X be a uniou of cylinders of rank 1 of whk'h iudcx 
i belongs to a sui»set. J of I, and let Dt := Bï. Dcfinc a function R • X  NU {} 
by R(.) = inf{n  0 • T G Ba } + 1 and for each « > 1. dcfine inductively 
= Y  T.- 
ç*3" TR()3 ". X denote X* :=, (U=o (N.>o{R(-r) > "})) and 
r := U {(,....) < r'- x,,.... ç ,,}. 
n>l 
Then it is easv to see that (T*, X*. Q* = {XL}6I. ) is an FRS Markov svstem. For 
= =o 5T (.r). 



2954 M YURI 

Definition. We say that an FRS Markov system satisfies local exponential insta- 
bility with respect to B if (05) " B 0 < 3'* < 1, 30 < F* < oc Sllch that Vn >_ 1. 
n--1 
er\", T'(n)=sup{diam}'l}'e V T*-'(Q*)} <_r*3' *n. 
j=0 
Definition. We say that a potential çb • X - IR satisfies local bounded distorsion 
with respect to B if there exists 0 > 0 such that ((16) • Vz_ = (iL .. • il*_l) E I*,  0 < 
Le(/) < oo satisfying 
Ici(ri(x))-d?(vi_(y))l <_ Le(i_)d(x,Y) ° (Vx.y e Tli-'Xi_) 

and 
sup Z L¢(ij+...ilgl) < 
/I* 3= 0 
Under the conditions (05-06), we can easily verify that {@*v •   I*} is an 
equi-H6hlr contim,ous fmnily (cri [19], [20]) and % Var(T*,ç*) < . Both 
conditions (05-06) can be eilv verified for all higher-dimensional examples in 8. 
In the test of this section we shall state relations between jump transformations 
associat.ed to B and induced maps over B. Let Rs " B   U {} be the first 
return function defined bv Rs(a) = inf{n k 1 • Tn G B}. Then we define the 
induced mapTB over {xG B " Rs(w) < oe} by Tsx = Tn()z and the induced 
Rs ()- Th(a)" Then 
potential B, " {x B • RB, (x < oe}  N by çBx (X) = 
we can immediately see the following facts. 

• +(0-¢oT,), 

Lemma 15 (Lemma 4.1 in [18]). Suppose that B consists #fall cylinders. Then for 
• -  isaTt3- 
any T-invariant probability measure m witb m(B) > O, mB .-- m(Bx) 
invariant probability measure and m* := mBT[[3 is a T*-im,ariant probability 
measure, m can be written in terres of m* by Schweiger's formula (see (3) in §5) 
and in terres of mt3 by Kac's formula (see Lemma 16). 

§2. TOPOLOGICAL PRESSURE FOR POTENTIALS OF ,VEAK BOUNDED VARIATION 
Let (T, X, Q) be a transitive FRS Markov system and let çb : X -- ]R be a 
potential of WBV. For each n > 0 and for each U  L/ we define the following 
partition functions : 

i:li_]=n,int(TX,, ):UDintX, x v£x=xcl[intX£) 

n--1 
exp[ Z (pTh(x)], 
h=0 

z,,(u, ¢) = 

n--1 
E sup exp[ Z dpTh(x)] 
i:[i_[=n,int(TX, )=UDintX, xX_ h=0 



COUNTABLE TO ONE MAIRKOV SYSTEMS 2955 

and 
z(u, ¢) = 
We further define 
z,, (,):= 

i_:li_l=n,int( T X ,, )=UDit X q 

inf exp[--" 6Th(x)]. 
xEXi_  
h=0 

n-1 
Z Z exp[Z 6Th(x)]" 
i_:li[=.int(TX,n)DintX,  vi_x=xEcl(intX_) h=0 
We shall define the topological pressure as thc asymptotic growth rates of these 
partition flmctions. 
Theorem 1 (%)pological pressm'e for I)otcntials of WBV). Let (T,X.Q) be a 
transitive FRS Markov system and let  be a potential qf Il'BI2 For each U G , 
lina L log(U, ), lim  log ,(l r, ), lim L log Z(U, ) exist and do 
hot depevd o r. Furtheore, the limits coincide with lim L logZ(). 
1 log Z,,() the topolo9ical pvssure for 
 call the limit Ptp(T. ) := lim,.oe  
. The next fa«t tan t)e verifie(l easilv. 
Lelnma 6. Uvder the next conditiom Z() coincides with the us'ad partition 
(1) For Zo  X,...i, with Txo = xo, either xo  intXi...i or .r0  clXj...j 
.for(j...j) ¢ (i...i). 
Let ç be thc filfite disjoint partition gcnerated by .  shouM claire that 
if a periodic poiut x0 with period , is COlltaincd in a cyliudcr Xi...i, satisviug 
Xi...i, C itl  for SOlne 1  G ç, then '0  OXi...i, If hot, we have a contradiction 
to Xo  intl  because of xo  T(OX...i,) = O(TnXq...i). By usiug this fact. we 
will see that all higher-dimensional exanlples in 8 satisfy (1). The Artin-Mazur- 
Ruelle zeta function @.e(z) is defined by @,e(z) = exp[, Z()]. Then the 
radius of convergeuce of Çr, e(z) is giveu by pe = exp[limsup  logZ()] -. 
ç define 
W(T) := {: X   I  satisfies WBV and Ptv(T,) < OE} 

and 

WB(T) := {6 e W(T) [ l'ar,6 --, 0 (,, --, 0),11611 := 
We can easily see that the pressure fulmtion Ptop(T, .) : W(T) --, 1I{ satisfies conti- 
lmity, convexity aud V6, 62 c W(T), Ptop(T, cb + 62) < Ptop(T. 6) + Ptop(T. 62)- 
Furtherrnore. by applying Theorem 2.4 in [7] we have the follov«ing fact. 
Theorem 2. î;B(T) is a Banach space and Ptop(T, .) : î;B(T) --  is a Lipschitz 
continuous convex function. 
Definition. If an FRS Markov system (T, X. Q) satisfies that VU c L/, BXi  Q 
such that Xi C U and T(intXi) = intX, then (T, X, Q) is called a strongly transitive 
FRS Markov system. 
Theorem 3 (Topological pressure and the spectral radius). Let (T,X,Q) be a 
strongly transitive FRS Markov system satisfying Lt f 12 ¢ . Let ç be a potential of 
weak bounded variation. Then VU  lg f3 l? and Vx  U, linloe 1 logL;lu(x) = 
Ptop(T, 6). Futhermore, lim,> 1_, log IIL;ël][ = lim_oe L 



2956 M. YU1RI 
\Ve can easilv verify ail conditions iii Theorem 3 for exalnples in §8. 

§3. THE CONSTRUCTION OF WEAK GIBBS CONFORblAL blEASU1RES 
Let (T. X. Q) lin a transitive FRS Markov svsteln and let  G (T). Suppose 
that there exists a union of fldl cvlinders B(C X) with respect to which (T. X. Q) 
satisfies local eXpOlential ilstabi}ity alld  satisfies local bounded disçortion. For 
pR(g)--I T h (£) We define 
the derived potential *(,r) = h.=0 
(]..4)Çl*n:it('F * Y )Dit., v]...nX=XÇcl{intXL..4n ) 
Then l»y Theoreln 1. Smlmmble variati(ms of * allow OlW to show that 
1 
 i,ll - g Z,,(*) := Pp(Y*,*) e (- l- 
Theorem 4 (A constru('lioll of confornlal lneasures via jmnl) transformations). Let 
(T, X. Q) bc z hmtsilite FR+" M«u'kof .sy8tem ittd lel  G }V(T). Suppose that there 
«.rist. « utiot «t( full c,lliuder., B(C X) u,ith re.spect o u,hich (T,X.Q) satisfies 
local e.rpotetttiM in.l«bilil9 ami  .ç«tti,sfies loco,[ boutded dislortion. .4s'zlte further 
lhat ]{£»*-nmin{a.&p(T*,ç*)} 1 Il < . T»«,, lhere emi.st.s a Borel probabilil m«asure 
 on X suppovled o X* .s.o, li.sfyitg 
As we bave amtounced in 0. for constructing a xxeak Gibbs measure for  of 
XVBV. we shall considcr thc following generalized Bowen's equation: 
The existence of a zero of the equalion (GBE) %llows Kom the standard argmnent 
in the case when 0  Ptop(T*,¢*) <  because of contimfitv of the function 
should notice that the uniqueness of thc zero of (GBE) follows Kom the "'strictly'" 
decreasing property of the fmtction, s  Ptop(T*, *-.sR) in the standard situation. 
Here we have no evidence of it although the funclion is decreasing. If Ptop(T*, *) < 
0. then mder the assumption Ptop(T*, * - RPtop(T*. *)) <  we see that 
Ptop(Y*, O* -- Ptop(*, *)) k Ptop(T*, * - Aop(T*, *)) = 0 
and so we can reduce to the previous case. If Ptop(T*. ¢*) = . then we cannot use 
the st.andard argulnent. Now we corne to state the next kev lemma, which allows 
one to establish Theorem . 

Lelnma 7 {The existence of a zero of (GBE)). (i) /f 0 _< Ptop(T*, çb*) < oc,, then 
Ptop(T, ) > 0 a,,d s0 _> 0 satisfgi,9 Ptop(T*, * - s0R) = 0. 
(ii) ff Ptop(T*, çb*) < 0 o,td Ptop(T*, çb* -/ï'Ptop(T*, çb*)) < oe, then Ptop(T, çb - 
Ptop(T*, çb*)) _> 11 a,,d s0 _> 0 sa, tisfgi,,9 Ptop(T*, (çb - Ptop(T*, çb*))* - s0/ï' ) = 0. 
(iii) /f sup{s G IR: Ptop(T*,c)* - sR) = oc} = lnill{O. Ptop(T*,O*)}, then s0 _> 
l,lil,{(), Ptop(T*. d)*)} .sttc[t ll-at Ptop(T*. d)* - s0) = 0. 



COUNTABLE TO ONE MARtxOV SYSTEMS 2957 

We recall the formal power serics (T,4(z) = exp[,= 1 -g- ,(¢)] which is called 
the Artil>Mazur-l:hmlle zeta fimction. The next product formula of zeta functions 
plays an important foie in proving Lenuna 7. 

Proposition 1 (cf. [17]). II'e tan write 

(2) 

çr,¢(e×p(-s)) = çr-,«-R(1) × 

Corollary 1. /.f s > Ptop(T, 0), then Ptop(T*, O* -- .s/-¢) <_ 0. 

Bv Theorelu 3, the assumption ]]£O*-Rmin{0,Ptop(T*,O*)}l]] < C implies either 
0 < Ptop(T*, *) < oc or Ptop(T*, * - r¢Ptop(T*, *)) < oo is satisfied. He,lce 
it follows froln Lemma 7 that Ss0 > nlin{0, Ptop(T*, *)} satisfying Ptop(ff*, * - 
a'0H) = 0 and II:«,olll < . Since Q* = {x,_}i¢** consists of full cylinders and 
sulnlnahility of variations ,__ Iar,(T*. * - s0/i') < oo is valid, we Call apply 
P. Walter's argmnent in [14] to show the existence (,f ail exp[soR - ¢*]-conforlnal 
measure with resl)ect to T*. 

Lelnma 8. There exisls a Bmvl probability mea.'nw oe on X solisfging £.* 
¢._soRlY : 
and v(intX*) = 1. 

In §9 we shall show the existence of an exp[s0- ¢]-conformal measure for thc zero 
s0 of (GBE) by using the conformal measure oe on X* and show s0 = Ptop( T, 
which implies uniqueness of the zero of (GBE). At the end of this section, we 
shall consider the case when I]£¢._tni{o.e,o(r.,¢.)}lll = oc. B.v Tlmorem 3. if 
* n 
Ptop(T , ¢*-&,]ï') = 0. then therc exists suflïciently large n such that ]] (¢-,o)* 11] = 

]]£(¢_,)11] < oc, where 
--1 

(0- So) := Z (- s°)*T* = 

n-1 

We shall introduce a new stopping time (depending on n > 1) defined Oll X* bv 

R,(x) := inf{k _> n I Xi...i(x)  

n--1 r--I 
V r*-Q*} = 2 

Then a new jump transformation S* detiued bv S*(x) := 7"R(X)(x) is equal to T *' 
and the next facts can be verified easily. 

Now we shall consider a two-parameter falnily of functions {(¢ - s) I (s, n)  
]R x N} and the equations Ptop(T *n, (¢ - 8)z) = 0. Applying Theorem 4 gives the 
next result. 

Proposition 2. Suppose that ail conditions in Theorem 3 are satisfied. If there 
exist .s0  ]R and no  N such that Vn _> n0, Ptop(T *n, (¢ -- S0)z) = 0. then there 
exists a Borel probabilitg mcasure oe o X supported on X* satisfyin9 -4-T[X, = 
exp[s0- ¢](Vi e I) ad p(UiFio.\i)= o. Furtherrnore. so = Ptop(T,¢). 



2958 M. YURI 

§4. INDIFFERENT PERIODIC POINTS AND NON-G1BBSIANNESS 
Let (T, X, Q) be a transitive FRS Markov system and let  E kV(T). The next 
lemma follows from the definition of Ptop(T, ) directly. 
1 q--I 
Lemma 10. Ptop(T, ¢) >_  h=o dpTh(xo)( VxO  X, Tqxo = 
Definition..r0 is called a generalized indifferent periodic point with period q with 
1 q--1 
respect, to  if Ptop(T, ç) =  h=0 (pTh(x0) • If x0 is hot indifferent, then we calt 
• fo a generalized repelling periodic point. 

If a potential  of \VBV admits a generalized indifferent periodic point, then we 
can observe interesting statistical phenomena. More specifically, if there exists an 
exp[Ptop(T, )- ¢]-conformal measure u, then t he above definit ions can be described 
in terres of the local .]acobians with respect to u, that is, 

d(uT q) 

q--I 
Ix,...,q(o)(xo) = exp[qPtop(T, dp) - Z cpTh(x°)] = 1. 
h=0 

Then we have the following facts. 

Proposition 3. Let .r0 be a generalized indifferent periodic point with period q with 
respect to c G kV(T). Let u b« an exp[Ptop(T,b)- O]-conformal m«asure. Th«n 
(i) Vs  1. Ptop(T, s0) = sPtop(, 0) and Vs < 1. Ptop(. sO)  sPtop(T. ). 
(il) u(Xi...i, (x0)) decaBs subeonentiall9 fast. 

Proof. Bv Lemlna 10, we have Ptop(T, sdp) > s  Eï dpTi(xo). In particular, if x0 
is a generalized indifferent periodic point for b, then Ptop(T, sdp) >_ sPtop(T, dp). We 
recall that by Lemma 2 the conforlnat measure u is a weak Gibbs measure for b of 
WBV. Then we bave for s > 1. 

1 1 

where both C, and Kn satisfv tim,_oe -1 ]ogC, = 0 and lilnn--,ec_1 logK, = 0. 
Since lilnn--,oe _1]ogCnIÇn = 0, we bave Ptop(T. sdp) _< sPtop(V, çb) for s _> 1. (ii) 
fotlows frolll Proposit.ion 6.1 iii [21]. [] 

Let us recall that u was obtained by constructing a jump transformation in 
Theorem 4. Then we can associa.te the generalized indifferent periodic points to 
the lnarginal sets Ç>_0 D. 
Proposition 4. Let xo be a generalized indifferent periodic point with period q with 
respect to dp  kV(T). 
(i) (Failure of bounded distortion) 
Cnq(XO) :-- sup exp[-ïq-l )Th(x)] - oc 
nq--1 
C'Xil ..... q(XO' exp[Eh_0 (bTh(/)] 
monotonicall9 as n --* 
(il) xo  Ç),>0 D,,. 



COUNTABLE TO ONE MARKOV SYSTEMS 2959 

Pro@ Since C,(Xo) is the distortion of d(vT') 
d over cylinders Xil...i(xo), (i) follows 
from Lemma 6.1 in [211. S,,ppose xo ¢ 0ao D. Then by Sublem,na A (see 9) we 
bave .ro  X*. Since  Vr,(T*, ç*) < oe ilnplies that C,q(Xo) cannot increase 
monotonically, we have a contradiction to (i).  complete the proof.  
Remark (C). We claire that 0 D, can contain repelling periodic points. 
If we have a T-invariant probability mea.sure p equivalent to u via Kac's formula 
(Lemma 16) or Schweiger's formula (3) in 5, then the invariant densities d/du 
are typically mfl)ounded at indifferent periodic poinIs with respect to u (Lemma 
6.2 in [21]) so that we tan sec interesting phenomena from a statistical point of 
view ([19], [21]). For example, under the existence of a generalized indifferent 
periodic point ,r0 with respect to , Ihe raie of decay of correlation may be slower 
than u(X..., (x0)), which decays subexonentially fast by (il) in Proposition 3.  
referee [21] for fllrther details. On the other hand, the Dirae measure m supported 
on the generalized indifferent periodic orbit with respect o  satisfies Ptop(T, ) = 
h.(T) + f. çdm. Hen«e if we can esIablish a variaIional principle for the topological 
pressure and tan construct a T-invariant measm'e p equivalent to Ihe weak Gibbs 
lneasure u for  wiIh -Ptop(T,), theu by Lemlna 3 we sec immediately failure 
of uniqueness of equilibriuln states. Furtherlnore, by the definiIion of indifferency 
we can show faihlre of Gibbsimmess of equlibrium states for ç with generalized 
indifférent periodic points. 
Theorem 5 (Characterization of non-Gibbsianness). Suppose that a potential ç 
with Ptop(T, ) <  admits a generalized indifferent periodic point x0. Then there 
is no Borel probabilit meas'ure that is Gibbs for . 

§5. EQUILIBRIUM STATES FOR POTENTIALS OF 'VEAK BOUNDED VARIATION 
Let (T. X, Q) be a transitive FIlS Markov svstem and let ]lIT(X) be the set of 
all T-invariant probability measures on (X,,T). For m E MT(X), I,n denot.es the 
conditional information of Q with respect to T-¢ -. We denote 
JIIT(X,c) := {m E MT(X) I I + 4)  L(m),either h,(T) < oc or 
.,,.O dru > -ec is satisfied}. 
Theorem 6. Let (T X, Q) be a transitive FRS ]llarkov systern ad let dp  W(T). 
Suppose that there ezists a union of full cylinders B(C X) with respect to which 
(T, X, Q) satisfies local ezponential instability and c satisfies local bounded distof 
tion. Let u be the exp[Ptop(T,¢5) - dp]-conforrnal measure supported on X*. As- 
sume further that F := Ç),>0 D, consists of periodic points. If fx* Rdu < oc 
and H,(Q*) < oc, then there ezists a T-invariant ergodic probability rneasu tt 
equivalent to u that satisfies the following variationai principle: 
dp) = bu(T) + .. dp dp = sup{ h,(T) + \. dp dru I m  ]lIT(X, dp) is ergodi«}. 
Ptop (T, 
If ET(X, cP) := {m  MT(X, cp) [ h(T) + fxcpdm = Ptop(T,¢)} contains at 
least two elements, then it iInplies ph.vsically coexistence of different phases, which 
is so-called "phase transition". Phase transition may be related to failure of Ihe 
Gibbs property of equilibrium states (see Theorem 5). 



29ô0 M. YURI 

Corollary 2 (Phase transition). i'e assume all conditions in Theorera 6. ff F 
consists of 9eneralized indifferent periodie points with respect to ch, then tbe set 
of equilibrium states for ch is the convex hnll of p and tbe set of iTvariant Borel 
probabilil9 meas,u«'es supported ou F. 
In order to prove Theorem 6. we need a sequence of lemmas. Let 3IT*(X*) 
denote the set of ail Borel probability measures on X* invariant under T*. For 
ch* - sR we define 
Mr.(X*,ch* - sR):= {m* G MT.(X*) [ either b,.(T*) <  or 

i.(çb* - sR)dm* > -: is satisfied}. 
Le s0 = Ptop(T,¢). Then P. Walter's lnethod in [14] can apply for T* and for 
* - soRso that there exists the unique equlibrium state ff* equivalent to u and 
the following variatiolm.l principh, is valid: 
0 = r(T*, O* - SoR) = h,,. (T*) + f (* - 
= ,p{,,,*  t.(x*,ç* -.t) I ,«(T') + [ 
(« 
s0R)dm*}. 
« 
Since ,, I "ar. (T «, 0* - 0R) < OE implies the i,ounded distortion property with 
respeçt to u • 
Slip Slip slip d(œeT.n ) < 
-- t Vj=o T*-(Q* d 
we can show ergodicity and Bowen's Gibbs property for p*. If fx* dp* < OE, then 
the next Schweiger's forlmfla ([13]) gives a T-invariant ergodic probability mesure 
p equivalent to oe that satisfies p(B) = (x-- dP*) - > 0" 

and bv Lemma 5 for f ¢ Ll(p), 

[ Ji\-. -î--(g)- fTi(x) d#* fB, --î=d ()- 
(3)* 
fdlt 
= fx. @* = f, ,d, 
(cf. Lemma 4.2 in [18]). Sin«e fç. Rdp* <  gives the equality 
and Hu. (Q*) < OE, gives b.(T) < ,, we can establish the following characterization 
of the zero so of (GBE). 

Lemma 11. /fp* e Mr- (X*) is ergodic and satisfies ht. (T*) +f\.. (* - «oR)dp 
= O, fx. Rdp* < cx and Ht.(Q* ) < cx, then 

S 0  

,..(T*) + L\-. O*dp* 

f\.. Rdp* 

= h t, (T) + f\ chdp, 

where p is obtained bg fornrula (3). 



COUNTABLE TO ONE MAFIKOV SYSTEMS 2961 

By Lemma 11 we have a T-invariant ergodic probability measure # equivalent 
to  that satisfics Ptp(T, ç) = b,(T) + fx çdl*. 
Lemlna 12 (Lelmna 4.4 hl [18]). g a T-invariant probabilitg measure m sotisfies 
re(B1) = 0, then F := n0 Dn is a full measure set with respect to m. 
Proof of Teorem 6. Bv Lelnma 11 for ail T-ilvalialt ergodic probability lneasures 
m on X with re(B1) > 0 and m  Iç(X. ¢), we can establish 
,(T) + £x- Ca,.- P(T. ) > 
0= 
/I(B1) -- (B1 ) 
On the olher hand, bv Lemma 12, any T-inval-iant ergodic probability ineasure m 
on X with m(B) = 0 satisfies re(F) = 1. In particular, if F := ,,0 D, consists 
of periodic points, then ,(T) + fx dm = J çdm  Pwp(T, ç), which completes 
the proof of Theoreln 6.  

§6. TtlE ('ONSTRUCTION OF O-FINITE ('ONFORMAL MEASURES 
VIA IND[X'ED MAPS 
Let (T, X. Q) be a transitive FRS Markov system and ç 6 çç(T). Suppose that 
(T, X. Q) satisfies local exponential instability and ç satisfies local bounded dis- 
tortion with respect to B1 : j Xj(ff C l). Let .4 = cl(iotXj) for j E J and 
put A := Uj .4.  define the first return function R4 " A   U {} and the 
induced map Ta over { A • R.4() < OE}. Bv the Markov property, there exists a 
partition of the set B A) = {.c ¢ .4 • Ra(.r) = k} for each k  1 so/bat T  restricted 
to the interior of each element of the partition is a homeomorphisln onto its ilnage. 
4 denotes the set of all indices corresponding to such elements of the partition 
of ,>1 Bi -A)" Then {v, : i ¢ IA } is a familv, of extensions of local inverses of 
. A()-- çTh(x) _ sR4(x). Bv 
For s Ç N &lld a'e U=I [:A) We defille *}2)(w) = h=0 
Lelmna 4 we can easily see the next fact. 
Lemma la. If each Ai C A satisfles TAs = X the, 
We recall the [ol]owing resu]t in [6]. 
Lm a ([6]). Z I1£,111 < . t   P»(«,é[2 ))  «ti,,, 
,,t{ e - P(4, é ")) e }. 
We suppose that 1 £O1]SSt8 of  sing]e fui] cylhder X and the [o]lowing con- 
ditions are satisfied [or A = 
(05)* 90<< 1,0<<,such that 
and there exists 0 > 0 such that 
(06)* V = (it...iN) E IA and all 0  j < I1,  0 < Lé(i+l...ill) < 
satisfying 
le(%+,...,. (z)) - ¢(v.6+,...., (u))l  L(i+ ...il)d(z,U)  (W.u e A) 



2962 M. YURI 

alld 
I1- 
sup E L4)(iJ+"'ili-I) < 
ilA 
Since the conditions (05-06)* allow us to establish the WBV propertv, of WA a'(»), by 
Theorem 1,  lim, 1 log Zn(¢}}):= Ptop(T4 , CA(»})' where 
n( »'): X X X 
" " I n -!n x:x 
Furhermore, (05-06)* gua.ramee equi-Halder comhmity of {)q :   ln} and 
E% '-(T,  ) < - " i lG,il < . th Gç», .C(A)  C(A). 
Theorem 7 (A construction of -finite conformal measures via induced maps). 
Let (T X Q) be a transitive FRS Markov sgstem and let   (T). Suppose tbat 
th«r« «xts a full cylind«r Xj e Q satis]ying (05)* and (06)* for A := cl(itA3). 
min{O.Ptoç(TA,OA II 
the 
(i) 3so e  with Ptop(4,¢ s°)) : 0 (a g«n«ralized Bowen} «quation); 
(ii) there eists a Borel probability measure oeA on (A.  A) with oeA({X  
A  RA(X) < }) = 1 satisfying E(»oUA = 
(iii) there exi.sts a a-finite measure u on X satisfyin9 £u = [expso]u and 
(iv) in particular, if u is finite, the. Ptp(T. ¢) = so. 

Proof of Theorem 7. B.v Lelnmas 13-14 and Proposition 1, we have the existence 
of s E IR for which Ptop(TA, ¢ )) = 0 and (d  adlnits an eigenvalue 1. Then we 
can apply the main theorem in [5] so that (i)-(iv) are obtained. [] 

The next result gives a criterion of finiteness of u. 

Proposition 5 (A criterion of finiteness of œe). Suppose that all assumptions in 
Th«or«m 7 arc satisfi«d. Thon œee(X) : fA exp[s0- ¢]dœeeA + 1. In particular, if 
infxeA ¢(x) > --oc, then u is finite. 

Proof of Proposition 5. Let I' = {/ [Xt C D N } • First we note the following for- 
mula of u, which was obtained iii [5]: 

Then we see that oe(X) is equal to 

 k 
k=l j_eIA.]jl=k+l v_(A) /=1 



COUNTABLE TO ONE MARKOV SYSTEMS 2963 

because of the fact t.hat Xi C TA(V/  I'). By conformality of V A this coincides 
with 

&:l jI.«,lJl:c÷l L (A) 

k 
exp[( Z 0T'- s0)(x)] exp[--0(A°)(z)]&,A(Z) + 1. 

§ï'. THE CONSTRUCTION OF MUTUALLY SINGULAR EQUILIBRIUM STATES 
In this section, we show the existence of mutually singular non-atomic equilib- 
rium states by using induced systems. 

Lenmla 15. Let (T, X. Q) be a transitive FRS Markov system and let ¢ C V(T). 
Suppose that there exists a sequence of full cylinder:s { " M 
Xi}i=I(M _< oe) that sati$fi«s 
(05)* and (06)*, infeA ' ¢(x) > -oc, and 

ffgA rnin{O'Pt°p('TAi 
for each Ai --- cl(bdX). Let Fo := X and for each i > 0 define indu.ctively Fi+I --- 
Ç)n=o T-n(F fq AC+l)(C F). [l'e assume that for each i > O. 
dF 
iCA,+I 
for the Borel prvbability measure 'r,A,+ on Fi  Ai+I obtained in Theorm 7. If 
M 
= F := F con.ists of periodic points, then ther exists a T-ivariant eryodic 
prvbability measure p equivalent to an exp[Ptp(T, ¢) - ¢]-conformal measure  that 
satisfies the following variational principle: 
Ptp(T. ¢) = h(T) + ,. ¢dp = sup{h(T) + f,. d,n , m  Mr(X, ¢) is eryodic}. 
The equilibrium state p for ¢ is not necessarily unique. 

Theorem 8 (Phase transition and singular equilibrium states). IVe assume all 
conditions -in Lemma 15. If F := I___/1 Fi consists of generalized indifferent periodic 
points with respect fo ¢, then there exists a sequence of ergodic equilibrium states 
{#i IM that are mutually singular and the set of equilibrium states for ¢is the 
J 
convex hull of {#}îl and the set of invariant Bord probability measures support«d 
on . 

Lemma 16 (Kac's formula). If f A lAd(oe[A ) < oe and PA is a TA-invariant ergodic 
probability measure equivalent fo oelA, then the next formula gives a T-invariant 
ergodic probability measure # equivalent to , • 

RA (z)-- 1 
#(E)/#(A) =/A Z 
i=0 

lE o Ti(x)d#A(X)(VE e 

Lernma 17 (Finite entropy condition). Suppose that for si := Ptop(TIr,_, 
r._,A,d%_,A, > --oe. Then H,r.«,_ ' (QrA,+,) < 



2964 M. YU1RI 

Proof of Lemma 17. Let A = Fi ç) Ai+l and s = si. Since we have V3_ E I4. 
œeA(Xj)_ =/a exp[¢})(vj(x))]dœeA(X)" - -- > exp[--s]j]_ evA)inf exp[¢)(x)], 
the bounded distortion for a(°) allows us to see that 
ŒEIA z - 
14 OE IA 
whcrc C is thc bomdcd distortion constant exp[D(1 -)-l(dimnX)°]. These 
incqualitics allow ont to cstab}ish 
H,,A ((A)  [--¢A )dt'.4 + log C + s  H 4duc < OE. 
IA Z 

ProofofLcmma 15. Since n__ Var,(T.4 
,v'A <  is satisfied, it follows from 
Theorem 7 that œA, on At satisfying £(s)FAl = F.41 for S = Ptop(T,) and 
A 1 
an exp[Ptop(T,) -]-conformal measure on X. Fnrthermore. bv Proposition 5 we 
see that 
U(Al) -- PAl" The bounded distortion allows Olm to obtain an ergodic TAI- 
invariant probability mesure PAl  PAl with a density dlt 41/doeAl awav froln zero 
(s) 
and infinitv. Furthermore, by [14] there exists an equilibrium state PAl for 
with respect to T4 that is ergodic. In particular, since H,A (Q_41) <  we have 
for 81 : Ptop(T, ), 
top(TAl, CÀï )) 0 l'"Al (1) + ¢81 )d'', ) hA, («1) + 
1 1 
for all a,- invariant probabilitv, measures mA1 e MT, (A, ¢(s))A, (cri [14]). These 
inequalities and Lcmma 16 allow us to bave a T-invariant ergodic probability mea- 
sure g  oe that satisfies lq(A) > 0. pi(F1) = 0 and 
(**) 0 = t,(A,)-(. (T) + .. (¢- 
81)dl) 
for ail T-invariant ergodic probability measures m  3IT(X. ¢) satisfying re(A1) 
0. (**) is equivalent to the inequalities • .s = h m (T)+ f.x- dp  h(T)+ f x- dm. 
On the other hand. any m e MT(X,¢) satisfying t(.4) = 0 is supported on F. 
In fact, since X = (Ui=0 T-A,) u (0(T-A)«). re(A,) = 0 and T-invariance 
of m give m(F) = 1. Thus the set of ail T-invariant probability measures m 
supported on F coincides with the set of ail T-invariant probability measures m 
with m(A) = 0. Since (Tr,,r. Qrl := QF) is a subsystem of (T,X. Q). we can 
apply the above arguments for the induced svstem (%nr,, A2  F. Q%nr, )- That 
 eX rn--1 
is, for s = Ptp(TIr,,¢ ) := lim  log :lfl=  ,=.æer, "Pth=O CT(x)] 
and for the associated potential WA:r, (x) = h=0 - . 



COUNTABLE TO ONE MARKOV SYSTEMS 2965 

our assmnptions allow us to establish the i»ounded distortion for 
'.4cr so that 
there exists a 4rl-invariant ergodic probability measure PAr that satisfies 
> h« («r) + 
CA 
2Fl 
for ail 4=r,- invariant ergodic probabi]ity ineasmes m.4=F  ITn=r (A2  FI, 
ç(s=) ). Let p be the T-invariant ergodic prolmbilitv measure supported 
AF 
arising ff'oto It.4p via Kac's formula. Then sz = ht,=(T) + f x dp2  h(T) + 
fx çdm for ail T-invariant ergodic InObability measures m  Mv(A. ¢) supported 
on F that satisfy m(F 5 4z) > 0. Inductively we have a decreasing sequence 
{s}, where s = Pp(Tr_,ç) and a sequ«nce of T-invariant ergodic proba- 
bilitv measures {Yi al 
. }i= such that pi is supportcd on 1" _l.tt,(I'i_  A,) > 0 and 
probability measures m G -lr(X,ç) supported on l'i-1 with m(I'i_  Ai) > O. 
Since yi(Fi-1) I and pi(f/) 0. 
= = {11'}i=1 arc nmtually singular. Filmlly. for 
every T-invarialt measure supported on F that consists of periodic points we bave 
s 2 ,,,(T) + ./v çd,,,. Since {s}e is decreaing, we complete the proof. 
8. APPliCATiONS 
In this section, we show some examples of transitive FRS Markov svstems to 
which our theorems 1-8 can apply. 
Exanple 1 (Brun's map [lai, []8], [0]). ter X = {(.,',,.r) e  "0 < 
1}. and let 
for i = 0.1, 2 where we put  = 1 and z = 0. T is defined bv 
T(Xl,X2) = ( z, z= ) Oll 
1 
T(,FI,3.2) : (x2 1 1) on 
Then Q = {X,}_ 0 is a Bernoulli partition and (0.0) is an indifferent fixed l)oint 
(i.e., det DT(O. 0) = 1). Since T is a continuous piecewise C 2 nmp and a(n) = 
n -1, all conditions (01)-(04) are satisfied and dynamical instal)ility is polylmmial. 
 e that ç = -logdet DT is i)iecewise Lii)schitz continuous so that ç is a 
potential of XVBV. Furthermore, since each periodic point is contained in a single 
cylinder the property (1) is satisfied. Define B = X1 U X2. Then T* satisfies the 
uniformly expanding property and a direct calculation allows us to establish (06) 
for ç = -log[det DT[ (see [18] for more details). Hençe we can apply Theorems 
1-8. In particular, we can see that Ptp(T, ç) = Ptp(T*, ç*) = O. 
Example 2 (InhomogelmOUS Diophantine approximations [13], [15], [16], [17], [19], 
[20], [21]). For the transfornmtion defined in the introduction (Example A), we can 
directly verify all conditions (01)-(04). In fact, we can introduce an index set 
I={(î) • a.bZ,o>b>O, ora<b<0} 
and a partition {X()" ()I}, where X()= {(a',g)  X" a = []- 



2966 M YURI 

we can veri-IÇ(b) _< log(1 +a( -2)) and a(n)= O(n -1) (see [15], [20]) (cf. [16], 
[19], [21]). Hen«e b = -logldet DTI is a potential of WBV. Since each periodic 
point is contained in a single cylinder, the property (1) in Lemma 6 is satisfied. 
Let D, be the union of cylinders of rank n containing indifferent periodic points 
and let B,, = D,_D,. Thei1 the junlp transformation T* • Ui= Bi  X defined 
by T*(x, ) = Ti(x, ) for (x,9)  B, satisfies exponential decay of dialneter of 
«ylinders (see [19]), and we can veriÇv the validity of (06) for ç = -log[ det DT[. 
Indeed, for  G I* with 1] = , L¢()  3/ 2 and so 
Il- oe 3 
EI* j=O n=l 
Hence we hae smmnability of Var,(T*, *). which allows us to apply Theorems 
1-8. 

Example 3 (A complex continued fraction [51, [12], [15], [22]). For the transfor- 
mation T, dcfined in the introduction (Example B), we define .k'«+ = {z G X • 
[1/z] = ,a +'m} for each ,, + m G I := {ma + n" (m.) G Z 2 - (0,0)}. 
Then we bave a countable partition Q = {X},e of X that is a topologically mix- 
ing Markov partition and satisfies (01)-(03). The inverse brmehes to T take the 
forin vj(z) = 1/(j + z), where j G I and the vj satisfy (04). Therefore the inverse 
branches of the nth iterate of the transforllmtion T" take the forin 
,t 1 
Ujl ..... in(Z)- p" + zp.-- and Ivj, ..... j.(z)l = 
q + zq-i ]q + zq-]  
where p = jp_ + Pn- and q = jq_ + qn-, n  1. and p_ = a, P0 = 
0 = q_ and q0 = a. If the string j .... ,j_ corresponds to a cvlinder that 
contains one of the indifferent points, but the longer string j ..... j corresponds 
to a sub-cylinder disjoint from the indifferent periodic points, then ujl,...,j n is an 
imerse branch of the jmnp transformation T* which is uniformly expanding. For 
¢(z) = -log [T'(z)], WBV and (06) are satisfied. Further details tan be found in 
[22] in which multifractal formalism was established by applying our Theorems 1-8. 

§9. APPENDIX- PROOFS OF RESULTS IN §§2 AND 3 
For the proof of Theorem 1, we first verify the following facts. 
Lenmla 18. (18-1) VUk E bl, Zn(Uk, c) > 0 for all n > S. 
(18-2) VU  ,Vn,m > S.+(U,)  (U,)(U,). (S.ubadditivity) 
(18-3) VU.Ut  U. 
z+»,+» (u, )  z(u,é)(c»,c») 
s»--I st --1 
Proof of Lemma 18. (18-1) follows kom Lemma 1 and emark (A) (18-2) %llows 
flore the definition of (E) directly. Since +s,+s(G,) is bounded flore 
below bv 
 inf exp[ 
zXt 
:l[=s.int(TX,, )=U .itXi 1 CUl 



COUNTABLE TO ONE MARKOV SYSTEMS 2967 

n-1 
x Z inf exp[ Z çflTh(x)] 
x¢Xj_ 
j:lj l=n,int( T X j )=Uk Dint X  1 h=(J 

X Z xXt 
t_:ltl=s ,int(TXt» )=U ,intXt I CU 
and the transitivity condition allows one to establish 

si--1 
inf exp[ Z 
h=0 

and 

(18-3) follows from the WBV property of ¢. 

Proof of Theorem 1. By (18-2) in Lelmna 18 a.nd the  BV property of ¢, both 
lim ± log(U,¢) and lim - logZ(U,¢) exist for each U  b/. Since 
-- 1 
Zn(U, ¢) _< Zn(U, ¢) _< Zn(U, ¢), by the WBV property of¢, lin,.._,o  log Z,(U, ¢) 
also exists and all the limits coincide. By (18-3) in Lemlna 18, it is obvious that the 
limit does hot depend on U. Noting min<-_<N Zn(¢,Uj) <_ Z,(¢) <_ ;=1Z,(¢,Uj) 
allows one to complete the I)roof. [] 

In order to prove Theorem 3 we first show the next result. 

Lemma 19. (19-1) VV e "12 and Vx • I; £,l-(x) _< _,u,_v Z(U, ¢). 
(19-2) Vx • Uk • H, L:;1u(x) > Z,(Uk,¢). 
(19-3) Z,(¢) >_ 
X()(1) Ç Q satisfles X()(1) ç U ad T(int X(1)) = int X. 

where 

Proof of Lemma 19. We first note tha.t 1T = Uxjcv Xj because of the Markov 

property of Q. Then for x • l 
assertion in (19-1): 

the following inequalities allow us to have the 

jI:X)CV Utld:xUt (ji2...in):TXin=Ut h=0 
Utld:UIDV (ji2...in):TXn:UIDVDX  h:0 

Ut L¢:Ut D V (ji...in ) :T X, =Ut DX 
<_  zo(v,ç). 
Ut lA:Ut Dt" 

n--1 
exp[ Z 
h-----0 



2968 M YURI 

For a" E Uk, we have the following iuequalit, ies, xvhich give (19-2): 

;lvk(*) = 

jI:X CUk (ji2...in):xTX n 

-> 

exp[- CTh(vji»..,.x)] 
h=O 
h=O 

exp[- ¢h(vji»..i,x)] >_ Z,(Uk, ¢). 
h=O 

By the WBV property of ¢, Z(¢) is |)ollnded fl'oln below bv 

Then (19-3) ff»llows frolll the Markov property and the strong transitivity. [] 

Prvof of Theorem 3. By (19-1,2) iii Lennna 19, we have for x E V C U, 

1 
lira sup - log £lv(x) < 
n --- oc 'Il 

lin, -1 logN( III&X n(Ul,¢)) = Ptop(T, çb) 
n--,oe ?l I</<N 

 log£lu(X) > Ptop(T. çb). Out aSSUlnption Uk ff 12 in,plies U = 
and lira inf,_ g _ 
1". Hence we bave the first assertion. The test of the assertions follow fronl (19-3) 
in Lenmm 19 ilnmediatel b.  

Iii order to prove Proposition 1. we need two sublemmas. 

Sublemma A (Lenuna 3.1 iii [17]). [>0 D, ad X* are positively T-invariat. 
Furtherv,o're, Um=l T*-m(Ç],>o D,) contains ro periodic points. 

Pwof. The result follows from the equality (4) • R(Tx) = R(a') - 1 (R(x) > 2). 

Sublemma B. Deflne 

T'(X, T):= {x X I 3(i...i,0  I" such thatvh...i,x 
7),(X*,T *) := {.c e X I 9(q ...i_,) e I *' such that v,,...iz = x}. 

Pro@ Since x  7),(X,T)ç X* visits B1 infinitely oflen, we Call find a point 
y G 7)(«\*,T *) for some I <_ n such that TJy = :r for solne j < R(y) and 



COUNTABLE TO ONE MAF/KOV SYSTEMS 2969 

l-1 
,=0 R(T*m(Y)) = n. By the property (4) the converse is also true. Since 
I-1 I-1 R(T*m y)-I 
m=0 m=0 h=0 
I--1 
E=o (T*) - 
=  (¢- 
h=O 

we have the rest of the assertion. 

rz=l mT)n(X,T)OX * 

Proof of Proposition 1. By Sublemma A we first note that 7),(X, T) = {7)n(X, T)ç 
fqn>0 D } t2 {P(X, T) C X* }. Then we see that (,¢(exp(-s)) is equal to 
çlo>o»,o(exP(--8)) exp[ exp[-ns] x  exp[n-1 CTh(x)]] • 
n=l x(X,T)X* h=O 
Define for  > 1 > 0 E* 
_ ,, := {v e ç,(X*,T*) = 
such that  R(T*'())= , and z = 
Then ç,(X*, T*) = U>t E*.t and ç,(X. T)X* = U<, E,,t. Bv. Sublemlna B we 
see t hat for z  X*, 
.--1 
h=0 

n-1 

We complete the proof. 

= 

ProofofLemma 7. (i) Bv Lemmas 13-14. we have continuity of the flmction 
Ptop(T*,¢* - sR) on int{s e IR I Ptop(T*,¢*) E IR}. Then the existence of a zero 
s0 > 0 of (GBE) follows from the standard argmnent. Since Ptop(T*. çb* - 80/) = 0, 
by Corollary 1 we have so <_ Ptop(T, ¢). If Ptop(T, ¢) < 0, then we bave a contra- 
diction. For (ii), replacing ¢ by ¢- Ptop(T*, ¢*) allows us to reduce t.o the case 
(i). (iii) Since the case 0 _< Ptop(T*, ¢*) < oc is covered by Lemma 7(i), we 
suppose either Ptop(T*, ¢*) = oc or Ptop(T*, ¢*) < 0. If Ptop(T*, ¢*) = o% then 
sup{s e IR" Ptop(T*, ¢* - sR) = oc} = 0. Hence Vs > 0, Ptop(T*, ¢* - sR) < oc. 
Since the function s --, Ptop(T*, çb* - s/) is decreasing and continuons on int{s  
IR ] Ptop(T*,¢* - sR)  IR}, we have lim-m Ptop(T*,¢* - sR) = oc so that for 
suflïcientlly small s > 0. Ptop(T*, çb* - 8) > 0. Oll the other hand, it follows froln 



2970 M. YURI 

Corollary 1 that Ptop(T*, ¢* - sR) < 0 for s > Ptop(T, ¢). Hence we have a zero 
s0 _> 0 of (GBE). le Ptop(T*,¢*) < 0. then sup{.s  " Ptop(T*, ¢* - ) = } = 
Ptop(T*, ¢*). The sanie argmnent as those for the previous case allows us to have 
a zero .% k Ptop(T*, ¢*) of (GBE). 
d(vD = exp[* -- s0R]vi. If 
Proof of Theorem 4. Bv Lemma 8 we have for i G I*, a - 
- • " d ,Z, exp[h=0 Thvi=...i.Z -- son]. Since the property 
(4) • R(Tz) = R(m) - 1 (R(m) k 2) implies a'-.  e I*, the equality 
du - du (ii3---iz)d(m'*=**")du 
now ,, to  tht v& c D,,  ,., = xp[O(,)- 0](V* e X*). O 
the other hand. we know that the above equality holds for Xi C B since i  
I*. Finally. we establish VX, e Q. ].x,(.r) = exp[s0- O(z)](Vz e X*). It 
follows from Lemma 2 and Theorem 2.1 in [18] that s0 = top(T, ). The sertion 
u(Ue 0X)(= u(U,e , &,(X))) = 0 follows from u(i,,lX) = 1. which is obtained 
bv Lemma 8.  complete the proof. 

1 EFERENCES 

1. J. Aaronson, M. Denker and M. Urbadski, Ergodic theory for lllarkov fibred systems and 
parabolic rational maps, Trans. Amer. Math. Soc. 337 (1993), 495-548. MR 94g:58116 
2. R. Bowen, Equilibrium states and the ergodic theory of Anosov diflfeomorphisms, Springer 
Lecture Notes in /Vlathematics 470. Springer, 1975. Mil 56:1364 
3. M. Denker, F. Przytycki and M. Urbafiski, On the transfer operator for rational functzons 
on the Riemann sphere, Ergodic Theory and Dynarnical Systerns 16 (1996), 255-266. 
97e:5819ï 
4. M. Denker and M. Urbafiski, On the existence of conformal measures, Trans. Amer. Math. 
Soc. 328 (1991), 563-587. Mil 92k:58155 
5. M. Denker and M. Yuri, A noie on the constructzon of nonsingular Gibbs measures, Collo- 
quiurn Mathernaticurn 84/85 (2000), 377-383. MR 2001k:37013 
6. M. Denker and M. Yuri. Partially defined in]in, te terated ]unctional systems, In preparation. 
7. D. Fiebig, U. Fiebig and NI. Yuri. Pressures and Equilibrium states for countable Markov 
shifts, To appear in Israel J. Malh. 
8. P. H. Hanus, R. D. Mauldin and M. Urbafiski, Thermodynamic formalism and multi-fractal 
analysis of conformal infinite iterated functional systems, Acta Math. Hungar. 96 (2002), 
27-98. 
9. Il. D. Mauldin and M. Urbaflski. Parabolic iterated ]unction systems, Ergodic Theory and 
Dynarnical Systerns 20 (2000), 1423-1447. MR 2001m:37047 
10. Omri Sarig, Thermodynamic formalism for countable Markov shifts., Ergodic Theory and 
Dynarnical Systerns 19 (1999), 1565-1593. M1R 2000m:37009 
1 l. Y. B. Pesin, Dimension Theory in Dynamical Systems. Contemporary Views and Applica- 
tions, Chicago University Press, 1997. MR 99b:58003 
12. M. Pollicott and M. Yuri, Zeta ]unctions for certain multi-dimensional nonhyperbolic maps, 
Nonlinearity 14 (2001), 1265-1278. MIl 2002h:37036 
13. F. Schweiger. Ergodic theory of fibred systems and metmc number theory. Oxford University 
Press, Oxford, 1995. hlR 97h:11083 
14. P. Walters, Invariant measures and equilibrium states for some mappings which expand dis- 
tances, Trans. Amer. Math. Soc. 236 (1978), 121-153. MI/ 57:6371 
15. M. Yuri. On a Bernoulli property for multi-dimensional mappings with finite range structure, 
Tokyo J. Math 9 (1986), 457-485. MR 88d:28023 
16. M. Yuri, On the convergence to equilibrium states for certain nonhyperbolic systems, Ergodic 
Theory and Dynarnical Systerns 17 (1997), 977-1000. MR 98f:58155 
17. M. Yuri. Zeta ]unctions for certain nonhyperbolic systems and topological Markov approxi- 
mations. Ergodic Theory and Dvnamical Systems 18 (1998), 1589-1612. Mil 2000j:37024 



COUNTABLE TO ONE MARKOV SYSTEMS 2971 

18. M. Yuri, Thermodynamic formalism for certain nonhyperbolic maps, Ergodic Theory and 
Dynamical Systems 19 (1999), 1365-1378. lkIl 2001a:37012 
19. M. Yuri. Statistical properties for nonhyperbolic maps with finite range structure, Trans. Amer. 
Math. Soc. 352 (2000}, 2369-2388. MP 2000j:37009 
20. M. Yuri, Weak Gibbs measures for certain nonhyperbolic systems, Ergodic Theory and Dy- 
namical Systems 20 (2000), 1495-1518. lk|lR 2002d:37011 
21. M. Yuri, On the speed of convergence fo equilibrium states for multi-dimensional maps with 
indifferent periodic points., Nonlincarity 15 (2002}, 429-445. MIR 2002k:37006 
22. M. Yuri, Multifractal Analysis of weak Gbbs measures for Intermittent Systems., Commun. 
Math. Phys. 230 (2002}, 365-388. 

DEPARTMENT OF BUSINESS ADMINISTRATION, SAPPORO UNIVERSITY. NISHIOKA. TOYOHIRA-KU. 
SAPPORO 062-8520. JAPAN 
E-mail address: yuri0oEath.sci.hokudai.ac.jp, yuri0oEail-ext.sapporo-u.ac.jp 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2973-2989 
S 0002-9947(03)03257-4 
Article electronically published on March 14, 2003 

STRONGLY INDEFINITE FUNCTIONALS 
AND MULTIPLE SOLUTIONS OF ELLIPTIC SYSTEMS 

D. G. DE FIGUEIREDO AND Y. H. DING 

ABSTRACT. We study existence and multiplicitv of solutions of the elliptic 
system 
-Au---- Hu(x,u,v) in Q, 
--A-v ---- -H,(x,u,v) in Q, u(x) = v(x) : 0 on 
where Q C N, N  3, is a smooth bounded domain and H 
We assume that the nonlinear term 
H(x, , )  I1  + v[ q + B(x, u, v) with lira R(x, u, v) _ O, 
whcre p Ç (1, 2*), 2" :: 2N/(N - 2), and q Ç (1, oe). So some supercritica} 
systems are included. Nontrivia} solutions are obtain. Yhen H(x,u,v) 
is even in (u,v), we show that the system possesses a sequence of solutions 
associated with a sequence of positive energics (resp. negative energies) going 
toward infinity (resp. zero) if p > 2 (resp. p < 2). Ail results are proved using 
variational methods. Some new critical point theorems for strongly indefinite 
functionals are proved. 

1. INTRODUCTION AND MAIN RESULTS 

Cousider the following elliptic system: 
-Au : H(x, u,v) in . 
(E) -Ai, = -H.(x, , v) lu . 
(x) = v(x) = 0 on 

where  C ]K N, N _> 3, is a smooth bounded domain and H •  x ]K 2 -- ]K is a C'- 
function. Here H denotes the partial derivative of H with respect to the variable 
u. Writing z :- (u, v), we suppose H(x, 0) --- 0 and H: (x, 0) -= 0. Then z : 0 is a 
trivial solution of the system. Iu this paper we discuss the existence of uontrivial 
solutions. Roughly speaking, we are ,nainly interested in the class of Hamiltonians 
H such that 
H(x.u,v)[u[P+[v[q+R(x,u,v) with lira R(x,u,v)=0. 
I Il » ÷ Il  
where 1 < p < 2* := 2N/(N - 2) and q > 1. The most intercsting results obtained 
here refer to the case when q >_ 2*, v«hich correspond to critical and supercritical 
problems. The case when q < 2* has been studied by Costa and Magalhfies [5], 

Received by the editors June 18, 2001. 
2000 Mathematics Subject Classification. Primary 35J50; Secondary 58E99. 
Key words and phrases. Elliptic system, multiple solutions, critical point theory. 

()2003 American hlathematical Societ 



2974 D.G. DE FIGUEIIEDO AND Y. H. DING 

[61 and Benci and Rabinowitz [31. See also Bartsch and De Figueiredo [21, De 
Figueiredo and Magalhàes [7], De Figueiredo and Fehner [8] and Hulshof and van 
der Vorst [11], where similar systems also leading to strongly indefinite functionals 
have been studied. However, only subcritical systems have been considered in those 
papers. 
Letting 2. = 2*/(2* - 1) = 2N/(N + 2), we assume that H(x,z) satisfies the 
following condition: 
(H0) therearep (1.2"), q (1 ) andr  (1, l+q/2.)suchthat, for all 
(., z), 
IH(x. u,v)l  0(1 +11 p- +[vl -1) 

a.nd 

IH,,(x,., v)l < r0(1 + lui p-1 + Ivlq-1). 
In ail hypotheses on 11(z, z) tho 7i's denote positive constants independent of (x, z). 
We note that if q < 2", thon 2. < q/(q - 1), i.e., q- 1 < q/2.. Hence, it is possible 
that q < -r < 1 + q/2.. However, if q > 2", then 7- < q. Furthermore, we relnark 
that 7- can be very large, if q is sufficiently large. 
In addition, we need distinct conditions on H corresponding to the ces when 
p>2, p<2orp=2. 
First. consider the ce when p > 2. In this ce, we assume the following three 
conditions: 
(H1) there are  > 2, u > 1 and Rt  0 such that 
H(x,z)u + H.(x.z)v 2 H(x,z) whenever Izl  R1, 
with the provision that v = g if q > 2: 
(H2) there are 2.(p- 1)    p and 2.( - 1) <  such that 
H(,z) OE l (ll  + Il ) -- 2 O all (,z). 
and  = q if q > 2*; 
(H3) H(,0, v) OE 0 and H(z,u,0)= o(11) as u  0 unifor,nly in z. 
ç prove the following results. 
Theorem 1.1. Let (Ho) be satisfied witb p > 2. If (Ht) - (H3) hold. then (E) bas 
at least one nontrivial solution. 

In order to provide some more transparent hypotheses under which the above 
result holds, we next present some conditions on H that are sufficient for (Ho), 
(Ht) and (H2) to hold: 
(H) there are p  (1, 2*) and q  (2, 3c) such that, for ail (x. z), 

and 

IH,,(x,u,v)l <_ "),0(1 + lui p-1 + Ivl §-1) 

IH(oe, u,v)l <_ o(1 ÷ Izl p-1 4- 

(H[) there are/ > 2 and RI OE 0 such that 
H=(x, z)u + Hv(x, z)v > ¢tH(x, z) 
(H) for p and q as above, 
H(x. z) >__ ")'1 (11 p + Ivl q) -- "r2 

whenever Izl >_ 

for ail (x, z). 



STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2975 

Theorem 1.1'. Let (HIc) be satisfied voith p > 2. If (H[),(H;), and (Ha) hold, 
then (E) bas at least one nontrivial solution. 
Theorem 1.2. Let (Ho) be satisfied voith p > 2. If H(x,z) is even in z and 
satisfies (H1) and (H2), then (E) bas a sequence (z,) of solutions with eergies 
I(z) := t (}(IVul  -]Vvl u) - H(z,z)), going fo oe as ,,  . 
In order to describe the other results, let «(-A) denote the set of ail eigenvalues 
of (-A,H()): 1 ( 2  3  "" ". 
We now consider the case whcn p < 2.  makc the following assumptions: 
(Ha) there are it G (1, 2), u OE 2 and 3 OE 0 (3 = 0. if q > 2*) such that 
H(x,u,v) 2 H(x,u,v)u+ H(x.u,v)v-3 for all (x,z); 
(Hs) there are a  (1, 2)and 5  (0, 1/2)su«h that H(x.u,v)  ?41u[ a -SAv 2 
for all (x, z)" 
(u) if q  2-, th« U,.(z, ),  ,[ -(v + ) fo 1 (z, ). 
With these assmnptions we have the following three results, for the cae when p < 2. 
Theorem 1.3. Suppose that (Ho) holds with p < 2 and q  2. If H(x,z) also 
satisfies (Ha) - (H), tben (E) has at least one nontrivial solution. 
Theorem 1.4. Suppose that H(x, z) is even in z and (Ho) holds with p < 2 and 
q  2. IfH(x,z) also satisfies (H4)-(H), then (E) has a sequence (z) ofsolutions 
with negative enewies I(zn) going fo 0 as n  oe. 
Theorem 1.5. Let (Ho). with p,q  (1, 2), and (Hs) be satisfied. Then (E) bas 
at least one nontrivial solution. If. in addition. H(x, z) is even in z, then (E) bas 
a sequence (zn) of solutions with negative energies I(zn) going to 0 as n  oe. 
Finally, we consider the ce when p = 2, which presents some sort of resonance. 
Assmne 
(HT) there exist b0 _< 0 < a0 such that Ro(X, z) := H(x, z) - (aou   + boy 2) = 
o(z] 2)  z  0 uniformly in x; 
(Hs) there exist «  (1, 2), aoe  la0, Oe)«(-A), such that R(x,z) := 
u(, )-  t« 10(, )1  (l+ll-+vV -) o n(, ) 
 lv-(1 + 
The position of the numbers a0, aoe, b0 with respect to the spectrmn «(-A) plays 
a very essential role in the next result. For that marrer, let i,j, k be nomegative 
integerssuch that Ai =min{A  «(-A) • A > a0}, A =max{A «(-A) • A < 
{j if a = a0, 
[= j+k-i+l if a > a0. 
Now we can state out last result. 
Theorem 1.6. Let (Ho) be satisfied with p = 2 and r < 1 + q/2. Assume tbat 
u(z,) i v« i  ad atifi () d (gs). Th (E) ha at tat o« pai of 
nontrivial solutions if  = 1, and infinitely many solutions if   2. 
The ces covered in Theorem 1.6 include some ymptotically linear systems. 
Such systems bave been studied in [5], [6] and Silva [13]. However, their results are 
hot comparable with the ones obtained here. 



2976 D G DE FIGUEIREDO AND " . H. DING 

We organize the paper as follows. In order to establish multiplicity of solutions we 
need some new abstract propositions on critical point theoD- for strongly indefinite 
functionals, which will be provided in Section 2. These propositions are based on 
certain Galerkin approximations, and we emphasize that the functionals do not 
satisfv the usual Palais-Smale condition. In Section 3 we study svstems that are 
_-uperlinear in the variable u. and prove Theorems 1.1 and 1.2. In Section 4 we 
consider systems that are sublinear in the variable u. and prove Theorems 1.3. 1.4 
and 1.5. In both Sections 3 and 4. the variable v can have subcritical grooEh as well 
as supercritical grooEh. Finally. in Section 5. we consider a special asymptotically 
linear svstem and prove existence of multiple solutions. 

"2. CRITICAL POINTS FOR STRONGLY INDEFINITE FUNCTIONALS 

Let E be a Banach space with norm [[-[[. Suppose that E has a direct sure 
decomposition E = E  @ E 2 with both E  and E 2 being infinite dimensional. Let 
P denote the projection from E onto E . Assume (e) (resp. (e)) is a basis for 
E  (resp. E2). Set 
X := span{e, ..... .en} + E 2. ,\' := 
and let (X) ± denote the complement of X m in E. For a flmctioIml I  Ci(E, 1) 
we set In := Il \,, the restriction of I on \-n- Recall that a sequence (z j) C K is 
saidtobea (PS) sequenceifzj  Xn n --. _-'c, I(z)  c and I' (--) Oas 
j  :. If an)" (PS) e sequence bas a convergent subsequence, then we sav that I 
satisfies he (PS) condition. 
Denote the upper and lower level sets. respectively, by la = {z  E " I(z) _> 
a}. I  = {:  E" I(z) <_ b} and I =/ ç I  (denote similarly (I)«, (I)  and 
/)). Wealsoset K = {z < E- I'(z) =0}. /Q = Kç/. K  =KçI and 
/C =/Q c/C . 
Proposition 2.1. Let E be as above and let I  C(E, IR) be even u, tth I(0) = O. 
In addition, suppose that. for each m Ç N. the conditions below hold: 
Ii) there is Rn > 0 such that I(z) < 0 for all - 
,12) there are r, > 0 and a, --  such that I(z) > a, for all z Ç (X"-I) ± 
with Ilzl[ = rn" 
113) I is bounded from above on bounded sers of X': 
I_ 0 if c >_ O. any (PS) sequen«e (:) has a subsequence along whi«h zn -- z 
Then the functional I has a sequence ( ck ) of critical values, with the property that 
Ck  -. 

Remark 2.1. This proposition is more or legs known if the condition (I_) is replaced 
bv the (PSt* condition (cf. [1], [9]), or bv the usual Palais-Smale condition, that 
is. any sequen«e (:k) C E such that [I(zk)[ <_ c and I'(zk) -- 0 bas a convergent 
subsequ«nce (cf. [31). 
Proposition 2.2. Let E be as above and let I  C(E.I) be even. A.sume that 
I(O) = 0 and that, for each m  N. the two conditions below hold: 
115) there are r > 0 and a. > 0 such that I(z) >_ ara for all :  X m with 
ri6) there is b >0 u'*th bm -- O uch that I{:) <_bru for all :  (X-) ±. 



STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2977 

Moreover, suppose that either I satisfies the (PS) condition for ail c > O. or that 
the condition below holds: 
(17) inf I(/Ç) = 0. and. for allc >_ O. any (PS); sequ«nce (z,) bas a subsequence 
alon9 which zn  z  Çc witb z = 0 only if c = O. 
Then I bas a sequence (ck) of positive critical values satisfying Ck  O. 
Proof. Let E be the family of symmetric, closed subsets of E{0}, and let  • E  
N U {0, } denote the Krasnoselski genus nlap. Set 
% := sup 

where 

rU := {A • r- A c X a,,d (A) >. + ,,,}. 
Fix m  N. The Borsuk-Ulam flmorem implies that A  (xm-) ± 7  for eaçh 
A • Z TM. It follows from (Ia) that 
inf I(z) < sup I(z) < b,. 
zA z( k'm- 1 ) / 

On the other hand, since 
and so, by (I5), we obtain 

Therefore, 
(2.1) 

çX, ) = n+m. one bas S,, " := OBr,f-)X,rî E E n , 
inf I(z) >_ ara. 

ara <_ c n _ 

A standard deformation argument, using a positive pseudo-gradient flow. yields the 
?72 OO ?72 
existence of a sequence (z)=, with z  X satisfying 
1 1 
[I(z) - cl  and , m 
? 71 
 «an sun,e that I(z)  cm as n  . So, (z.,) is a (PS) sequen«e with 
(2.2) a  c  b. 
Now, if we assume that I satisfies the (PS)gcondition for c > 0, then the conclusion 
m 
follows. Next, suppose instead that (I7) holds. Then, along a subsequence, z 
z as n   with I'(z) = 0 and 0 < I(zm)  Cm. Finally, by (2.2), 
I()Sb o, 
and the proof is complete. 
Proposition 2.3. Let E be as above and let I  C (E,N) be even wth I(O) = O. 
Suppose, in addition, that the three conditions below hold: 
(Is) the «e e N .,  > 0 h tht I() 
(I9) there is b > 0 such that sup I(E 2)  b: 
(Ira) any (PS), c > O. sequence (z) bas a subsequence along which z 
Çc and Pz  Pz. 
Then I bas at least one pair of nontrivial critical points if  = 1. and infinitely 
many ctical points if f > 1, with positive critical values. 



2978 D. G. DE FIGUEIREDO AND Y. H. DING 

Pro@ Let E, % E T and c be as in the proof of Proposition 2.2. As before, by 
(I8) and (I9), we obtain 
" < b for all n • N and tri = 1,.-- ,. 
and we find sequences z G X such that, going to subsequences if necessary, 
I(z)  c and I,,(z n )  0 as n  , with 
b  Cl  C2  "'"  Ct 2 a. 
Using (I0), we can assume flrthermore that n'  z G « for m = 1,--- , g, as 
n  . If g = 1 the proof is complete. 
Consider (> 1. Let F = {z G " I(z) > 0}. aregoingtoprove that F 
is an infinite set. Arguing by contradiction, we suppose that F is finite. Çhoose 
0 < g < a  b < u satisfying 
tt < inf I(F)  sup I(F) < u. 
Let k G N be so large that 0  A := QF, where Q • E  X  denotes the 
projection. Then A is also finite, and (A) = 1. By the continuity of % for all 
5 > 0 small, (N(A)) = (A), where N(A) = {z  X  • dist(z,A)  5}. Set 
c = (.4)(x) . Sn« (A) c c ,,d Q "C  V(A). t onow rom th 
properties of  that (C6) = (N(A)).  remark that Q = p1 + (Qk _ p) and 
that the range of Q - p1 is k-dimensional. So bv virtue of (I10), we conclude that, 
for all c  0, any (PS) sequence (zn) has a subsequence along which zn  z G « 
and Qkzn  Qkz. Hence there are n0 G N and a > 0 such that for all n  n0, 
[]I,'(w)l ]  a for all w G (I)  C, 
where C = C6  X. Bv a standard deformation argument, we can then construct 
a sequence of odd homeomorphisms q • X  X such that 
,, ((&) k C?) C (Z,) 
(cf. [12]). For n0 sufficientlv large, we can suppose that 
p<% % _'"_c n<u forallnn0. 
Let G G E be such that inf I(G) > ( + c)/2. One then h 

and 

(,»,(c\c')) = (c\c') >_ (c) -(c') 
>  + t. - ")'(Ce) > r, + t. - . 
Thus rl,(G\C') • Z -1 and v <_ inf I(rl,(G\C') ) <_ %t-,. One finallv, con,es to 
t--1 
r, _< % < r,, which is a contradiction. [] 
From now on we turn to the system (E). We denote by I" It the usual Lt(Q) norm 
for ail t • [1, oc]. For q > 1 let Vq = H(Q) if q <_ 2* and Vq = H(Q) çlLq(Q), the 
Banach space equipped with the norin IIvllvç -- (IVvl ÷ IriS)1/2, if q > 2*. Let Eq 
be the product space H(t2) x I with elelnents denoted by z = (u, v). We denote 
the norm in Eq by Ilzll« = (Iwl ÷ Ilvll) 1/2.  has the direct sure decomposition 
Eq = E @ E +, z = z- + z + 
where 
E-={0}xVq and E +=H(t2) x{0}. 



STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2979 

For convenience, we will write z + = u and z- = v. t/eca.ll that by (A,),EI we 
denote the sequence of eigenvalues of (-/N,H{II(H)). Let e,, le,12 = 1. be the 
dgenflmction corresponding to A, for each n E N. Clearly, e + := (e,, 0), r E N, is 
a basis for E +, and e = (0, e,), n  N, is a basis for E-. 
Suppose that the assmnption (Ho) holds. Then 

(2.3) 

So the functional 

(2.4) 

(2.5) 
and 

H(x,z) <_ c(1 + lu[ 2. + I¢,1 ) o n («,z). 

il £ 
I(z) :=  ([Vul 2 -IVvl 2) - H(x,z) 

(2.6) ] H,,(x,z)b  ./o H,(x,z)b for ail 
By the Sobolev elnbedding theoreln and using interpolation, we obtain that 
u  u in L t for t  [1, 2*) and v  v in L t for t  [1, q). Noting that 
]H(x,u.v)  %(1 + u -I + ]v -) with 2.(r- 1) < q, (2.5) follows easilv since 
u in L ,v vinL 2.(-1) andç H() C L 2.. Next weseethat (2.6) 
is clely true when ¢  L . In general, for a ¢  t we proceed as follows. Let 
ç e L oe with   ¢ in L q as m  oe. So 
IZ(H(x,z) - H,(x,z))¢I = I£(H,(x,z) - H,(x,z))( + (¢- m))  , 
and using (Ho) we see that this expression is less than the following sure: 
+ , (> - ml + >vlm -', + va-m - 
which by its turn is estimated bv 
since (z) is bounded in Eq and L  is dense in Lq. So (2.6) is proved, and it follows 
that 
I'(z)w + I'(z)w for ail w  Eq 

is well defined in Eq. Moreover, I ¢ Çl(E'q,]l), and the critical points of I are the 
solutions of (E). 
Lemma 2.1. If (Ho) holds, then I' is weakly seqttentially cotinuous, hat is, 
I' ( z, ) -- l'(z) provided z, -- z. 
Pro@ If q < 2* this statenlent is well known. Assume now hat q > 2". Let z, -- z 
in Ev Çlearly, for all w = (ç, )  Ev we bave 
So it renmins to show that 
[ H=(x,z,)o -- [ H=(.r,z)o for all qO  Ha(t ) 
J 



2980 D.G. DE FIGUEIREDO AND Y. H. DING 

3. THE CASE p > 2 
Throughout this section let (Ho) be satisfied with p > 2, and assume that 
and (H2) hold. Observe that, by (H2), there exists R > () such that H(z,z) > 0 
whelever Izl > R. This, jointly with (H1), implies 
(3.1) H(x,z) > «(lui" ÷M)-c2 foran 
(see [10]). This, together with (2.3) and (H2), shows that 
(3.2) u <_ q and /3 _< q. 
Moleover, by virtue of (3.1) and (H..,), we lnay assulne, without loss of generality. 
that (since p > 2) 
(3.3) « > 2. 
1 e 2 e + for ail n (5 N. So 
Now we set. E 1 = E-, E 2 = E + and e n = , e n = 
Eq = E1 ( E2. Consider the flmctiolml defined by (2.4), which has the properties 
stated iii Section 2. 

Lemma 3.1. ,4ny (PS) sequence is bounded. 

Pro@ Let z, G X, be such that 
I(z,)  c and I'(z,)  O. 
--   n), we 
Cae 1: q < 2. h thi ca E = (n()) . m" (n), for ' := ( , 

have 

(3.4) 

=(_71 _ 1, )lVul ÷ ( 1 _ __ )lVvl 

÷  (;H"(x'zn)un + 1--Hr(x'zn)vn - H(x'zn)) -cloe 
1 £ IwI 1 1)lwl-c» 
->(- ) +(;- 
If q < 2. then (3.2) shows that pe < 2, and so I1=11 -< e(a ÷ I1=11), which implies 
that (z,) is bounded in Eq. Assulne q = 2. hvokillg (3.2), we get u  2. and so 
I  c(1 + [lz[lq) by (3.4). Since ti(x,z) > 0 for all [z] large, and 
one sees that [z[[ N c(1 + [[zllq). Hence, (z) is bounded. 
Case 2: q > 2. Note that in this case  = p > 2 in (H). So 

(3.5) 

I(z,)- -I'(z,)z = (7_H(a',z)z- H(x,:)) 
# - 1) f H(x, z) - 
_>( 
C, 

which, together with (H2), yields 



STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2981 

Using (Ho), we get 
(3.7) 
Next we estimat.e the integrals in the right, side of (3.7). Since 2.(1»- 1)    p, 
we have that 0 := /(1 + a - p)  2*. Using tho H61dor inequality, the Sobolev 
embedding theorem and (3.6), we obtain 
Similarly, since z - 1 < ffC., we bave 1 < w := /(1 + [3 - z) < 2*, and hence 
 Il-ll[  I' - 
Therefore, usiug th  est.ima.l«  in (3.7), wc oh/aih 
_ , l+(p-)/ 
Iç,t < «(1 + It  + IIzll+<-')/n) • 

(3.s) 
Since 

IVv,] = -l'(z,)(O, v,) - fa H-_(a', z,)z, + fa H,,(x,z,)u,, 
and using (3.5) and the above argmneuts, we obtain 
(3.9) IV,l 
Recall that, in view of out sumptions, (p- 1)/  1/2., (r- 1)/ < 1/2.. and 
fl = q if q > 2*. Hence, it follows from (3.6) and (3.8)-(3.9) that (z) is bounded 
in Eq. 
Lemma 3.2. Let z,  X,be a (PS) sequence. If q  2", then (z) contains 
a convergent subsequence. If q > 2", then te is a z  Eq such tat. along a 
subsequence, z  z and l'(z) = 0 and l(z)  c. 
of. By Lemma 3.1, (z,) is bounded.  can assmne that z,  z in Eq, z  z 
in (L()) 2 for ail 1  s < 2*, and z(x  z(.r) a.e. on . It follows from the 
weak sequential continuity of I' (see Lemma 2.1) that l'(z) = 0. Since I'(z)  0, 
we obtain 
). 
Using (H0) and the H61der inequa.lity, we obtain the estimate 
where  is as in the proof of Lemma 3.1. Hence I1  Iul, -hih implies 
  



2982 D. G. DE FIGUEIREDO AND Y. H. DING 

in Eq for ail z E Eq. Moreover, using again (Ho) and the H61der inequality, we 
estimate 
_< c([ç -- Pnçll + [n[ç-ll ç - Pç[ç+ ]ç[ç-l[ t' - Pat'[q) + 0. 
On the other hand, 
I[(z)(O,v - Pv) + [. H,,(.,z)(v- Pv) 
VV)L 
+ 
Lebesgue's theorem and the weak sequential continuity of H:(«,-) (see the proof of 
Lemma 2.1) yieht 
] Vv, - liln sup ] Vv ] = linl inf ( £ H: (,r , z ) zn - £ He ( x , z ) Z )noe noE  0, 
i.e., IVvl 2 lira sut,,, V*'I. This, together with the weak lower semicontinuity 
of norms, implies Vvu + [Vvlu. So v + v in H(). 
Therefore, if q  2*, we obtain that, along a subsequence, z + z in Eq and 
consequently I(z) = c. Next a.ssume that q > 2*. Observe that 

hence, 

Lebesgue's theorem then yields 
I(z) -c= liif  H(x,z,O - £ H(x,z ) >_0, 
that is, I(z) >_ c. 
Lemma 3.3. If(H3) also holds, there are r,p > 0 such that inf I(OBrE +) > p. 
Pro@ By (Ho) and (H3), for anv e > 0. there is c« > 0 such that 
H(.. . 0)< el,[ 2 + c«M 2". 

Hence, 

1 
and the conclusion follows easily. 
Let e E E + with IVel = 1, and set 
Q= {(se, v) • 0 _<  _< q, Ilvll <_ r2I. 



STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2983 

Lemma 3.4. If(H3) also holds, there are rl,r2 > O, wzth rl > r, such that I(z) <_ 0 
for all z Ç OQ. 
Pro@ By (H3), I(z) <_ 0 for all z (5 E-. By (H2), 
I((se, v))< 2 [7v]-c1 (lel÷lvl)÷c2 
The conclusion follows since  > 2.  

We are now in a position to prove Theorem 1.1. 

Proof of Theorem 1.1. Lemmas 3.3 and 3.4 soEv that I has the linking geometry. 
Let Qn := Q 0 Xn, and define 
ch := inf maxI(3,(Qn)), 
where Fn := {? e C(Q,X,) : /IOQ, =id}. Then p_< c, _<  := supI(Q). A 
standard deformation argument shows that thcre is z E X such that II(z)-cl < 
lin and I{I.(Zn)[I _< l/n. So we obtain a (PS): sequence (Zn) with c e [p, c]. 
Lemma 3.2 implies z -- z with I'(z) = 0 and I(z) >_ c. The proof is complete. [] 

We now consider the multiplicity of sohltions using Proposition 2.1. 

Lemma 3.5. I satisfies (Il)- 
Pro@ Using (H2), we obtain 
I(z) - z < lVul -  1]Vv] -c  (lui a + Iv] ) + 
C 2 . 
Since all norms in span{e,.-- , e,,} are equivalent, we obtain 
I(z) <_ -(«31Vu] -2 )lVu]- (-]Vv]-- «l]V]) --C 2, 
for ail z = (u,v) G X TM -- span{el,-.-,e} x 1. So (It) follows easily. 

Lemma 3.6. I satisfies (I2). 
Pro@ Since (X) ± C H(ft) and H(ft) embeds compactly in LP(ft), we have 
that r > 0 and r  0 as m  o, where 
(3.10) rh := 
sup ; 
see Lemma 3.8 in [14]. For z = (u, 0)  (X') ±, it follows fi'om (Ho) that 
 1 iÇtl2 2 _ ci itl p _ C 
I(z) = 1 IVl- H(x,,O) _> - 
1 
-2 
Setting r, = (pcr) 1/(2-») and a, = (p - 2)r/2p - c2, we corne to the desired 
conclusion. [] 

Proof of Theorem 1.2. Since H(x, z) is even in z, I is even. Lemma 3.2 shows 
that I satisfies the assumption (I4) of Proposition 2.1. Lemmas 3.5 and 3.6 show 
that (I) and (I2) hold. Clearly (I3) is also true. Therefore by Proposition 2.1, 
there is a sequence (z) C Eqsatisfying I'(z) = 0 and I(z,) -- o. The proof is 
complete. [] 



2984 D. G. DE FIGUEII:IEO AND Y. H. DING 

4. THE CASE l» < '2 
Throughout this section we assume that (Ho) is satisfied with p  (1. 2). 
also suppose that (Ha) - (H) hold. 
Let Eq = E 1 • E 2 be as in Sectim 3. Consider the flmctional 
Lelllllla 4.1. Auy (PS) scqu.«m'e (Zn) ha.s a subsequence collverging wea£1y to a 
criti«al poi'nt z qf J with .I(z)  c. a,d z = 0 only if z=  (1 i Eq. 
Pv@ The l)r()of is divi(h,d ino wo parts. 
Pat 1. The sequ(,n('c (z.) is bom.led in Eq. By (Ha) it follows that 
Hence IVu,,l  c(1 + IIzllq). If v > 2, we also got IV,l  c(1 + llzllq). If v = 2, 
wc uso (fla)and tire fa«t that Içl OE a,ll i, or,ie to obtain 
Thus, if q N 2*, thon (z.) is bounded in Eq. Assmne next that q > 2*. It follows 
ri'oto (Hi) t hat 
in Eq also in the case when q > 2*. 
Part II. %% can now suppose that z.  z in Eq, On  Z in (L(ll))  for all 
1 N s < 2*, and z(m  z(m a.e. in m Q. It follows that z is a critical point of 
J. As in the proof of Lonnna 3.2. using (H0) and 

Using (Ho), we have 
/-/,, (.r. 

+ fa Ho(x, z.(v - P. 



STRONGL INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2985 

Colsequent ly, 
(4.2) ('v, (v-- v,))L, = i Hr(.'r, Zn)( -- v) + o(1 ). 
Thus if q < 2", it follows from (4.2) that {Vt,[  
and so zn  z. This proves lhat .l satisfies he (PS);condition in this case. and 
that J(z) = c. 
Consider next q k 2*. The weak sequential contilmity of H,,(.r,-) (see the proof of 
Lemma 2.1) yields fn H(.r, zn)v  j; H,(.r,z)v. By (H), fn(z):= H,(.r, zn)t' + 
76(1v1+11 )  0. Using the fact that [v]  ]v[ and [t,]  1[, and applying 
Fatou's lemma to the sequence (f), we get 
Using this estilnate in (4.2), we obtain that IV,]  lilnsUp,,oe V'I, whi«h 
implies that. t,  ' in H). In Ol'dm to Colwlud« that .lz)  c, we use the 
est ilnat.e 
.l(z)-.J(z) =  (H.r, zn)- Ha',z)) + o(1), 
and so z  0. 
Remark 4.1. In a similar way, using even silnpler argulnents. OlW checks that. if 
(H0) holds with p, q  (1, 2), .1 satisfies the (PS) condition for all c. 
Remark 4.2. Let ,], = J x- denote the restriction of J on X . As in Lemlna 4.1, 
it is hot difficult to check that. if the sequence (z) Ç Eq, with z G X , satisfies 
J(z)  c and (z)  0 as m  , then it possesses a subsequence converging 
weaklv to a critical point z of J with J(z)  c. and z 
also bave, as in elnark 4.1, that. if (H0) holds with p. q G (1, 2), then anv such 
sequence bas a convergent subsequence. 
Lemma 4.2. There is an R > 0 such that J(z)  0 for all z = (v,O) with lzll 
ool. y (,,), w hv (,,..0)  «(1 + ). 
and the lelllllla follows, SillCe p < 2. 
Lemma 4.3. Fore > 0 small the is p > 0 such that J((ee.v))  p for all 
v  . whe el is the eigenfunction correspodin9 to the first eigem, alue  of 
(-.  (t)). 
Pro@ Bv (H.5), for e > 0 small. H(z, ee,v) k 24eeî- akv2: hence, 
, 1 'V'" -  > ('[e" -  '-)" 
J((ee.v)) = H(.r, ee. v) +  _ 
The conclusion follows. 



2986 D.G. DE FIGUEIIREDO AND Y. H. DING 

are llOW ready to prove Theoreln 1.3. 

Proof of Theorem 1.3. Recall that X TM  span{el,--- , e,} x l,q, and consider the 
restrictions J, as defilmd iii Rema.rk 4.2. Set Dn = Bn ç E 2 = Bn fq (HH(12) x {0}) 
and Dru = DR fq X m, where R > 0 cornes from Lemma 4.2. Define 

ç,n := inf max J(7(D,n)), 
where Fro := {7 • C(D,, S') : 7(z) = z forall z  OD,,}. It is well known 
tiret 7(Dru)  II" ¢ ¢ for all 7  Fro, where II" = {(«o.0)} x 1 with « > 0 
slnall. Invoking Lemma 4.3, we fix an ê > Il so snmll that there is p > 0 satisfying 
inf./(ll')  p. Then we have 
t'  c.  b := max J(Dn). 
The well-known saddle 1,oint lheorem (cf. [12] or [4], [14]) implies that there is 
z, G X m satisfying ]J(zm) - c]  l/m and ]].[(zm)]]  1lin. Now by virtue of 
Remark 4.2, along a sul,sequence, z  z with ,l'(z) = 0 and z ¢ 0, ending the 
proof.  

"Ve llOW turn to the proof of Tlworelns 1.4 alld 1.5. 

Lemma 4.4. If. in addition. ")'3 = 0 in (Ha), then J satzsfies (I5)- 

Proof. It follows from (Hs) that 
J(z) > c I I + ( 
(4.3) 
Since a < 2, the result follows in the case when q  2*. Next consider q > 2*. 
Suppose (I) does hot hold. Then for any r > 0 there is a sequence zj  X TM such 
that IIzll =  al,d J(z)  O. It fonows frein (.a) with z = z, and for r slnall. 
that Içl=  0 and [çvj[=  0. Ail this ilnplies that 
sumption (H0) and t he fact t hat (ai) lies in a finite-dilnensiolml subspace, it follows 
that fH(x, zj)uj  O. Colasequently, by (H4)with 73 = O, f Hv(x, zj)vj  O. 
This, jointly with (Ho), yields 
0. 
Hence, zj  0 in Eq, which is a contradiction. 

Lemma 4.5. J satisfies (16) 

Proof. By (Ho), H(a:.,,,0) <_ c(lu[ + I**l), alld so, for u • (X'-I) ±, Olle hg8 
 (Cll']p- [ 
l[v[œ)lVu[ = + (Cl  _ 1 
where Vin was defined by (3.10). Let b := 
Then 0 < b  0 and J((u,0))  b for ail (a,0) 



STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2987 

Proof of Theorcm 1.4. Since H(x,z) is even in z, d is even. If q < 2", then d 
satisfies the (PS) condition for all c (see the proof of Lemma 4.1). If q > 2*, then, 
using assumption (H4) applied to a critical point z, we obtain 

3(z) = 3(z)- £(z)(**,-v) > 1 1 Iwl + 

0. 
_'2 ,t  - 

This, jointly with Lennna 4.1, shows that (If) is satisfied. It follows from Lennnas 
4.4 and 4.5 that J satisfies (I5) and (I). Therefore, the desired conclusion follows. 

Finally, we prove Theorem 1.5. 

Proof of Theorem 1.5. The proof of the existence of one nontrivial sohltion is sim- 
ilar to that of Theorenl 1.3, using Ptemark 4.2 and LellllllaS 4.2 and 4.3. The 
other conclusion tan be obtained along the lines of the proof of Theorell 1.4, using 
Remark 4.1 and Lennlms 4.4 and 4.5. [] 

5. THE CASE p = 2 

hl this section we alwa,vs assume that (Ho) holds with p = 2 and r < 1 + q/2. 
We also suppose that (H7) and (H8) are satisfied. We will apply Proposition 2 3 
in order to prove Theorem 1.6. Thus, set 

/72 =span{eï, --- ,e} : /I ----spall{el,'--,et-} X Ç, /71 - /q(E '2, 

and 

X £ /71 (t)span{ï,-- + (7,"" (;} 
 • ,ek, , • 

2 + forl<< 
1 + for n E 1, and e n= en+i_ 1 
One may arrange the bases as % = %+ _ _ 
2 + for g < n < f+i- 1, and 2 - 
2 - forg-j<n<t e . %=%_t %=e,_ 
g--j, e n = en_t+ j -- -- 
for rt > f + i- 1. Consider the filnctional I given by (2.4). 

Lemma 5.1. I satisfies (Is); that is, there exist r, a > 0 such that I(z) >_ a for all 
z e X t with Ilzll = - 

Pro@ Let z = (u,v)  X e. Since v Ç span{el,.-- ,ej}, we have v Ç L . By (Ho) 
and (H7), for any e > 0, there exists ce > 0 such that 

/0(, ) < ell 2 + c,(M 2" + Iris). 

Thus 

1 
I(z) =  (IVul - a, olul ) - - ([x7vl - b01vl) - J 
Ro(x, 
Z) 
'(_ )  + ( ),-«- (: + 
> _ ao V 2 1 -bo 
-2   

Now the conclusion follows easily. 

Lemma 5.2. I satisfies (I9); that is. sup I(E 2) < oc. 



2988 D.C. DE FIGUEIIIEDO AND Y H DING 
Pro@ For z E E 2 we ha'«', using (Hs), that 
 IV'loe- f/(', z) 
[() =  
- '2  
which ilnplies that l(z)  0 for ail z e E 2 with IIz[[«large. 
Lenlma 5.3. Let c > O. Thcn any (PS)c sequewe ts bounded. 
Pv@ Wc decompose H] (t)  
H(t)=U-6 +, t,=,-+u +, 
«hcre - = span{e,---,eh} and U + is the orthogonal complelnent of U- in 
,l(t). 
Let (z) bc a (I'S)g sequence. Using the expression of I,': 
  +2 
plus (Hs) and the Hi;lder inequality, we obtail 
a--1 + 
A+ - 
where r = q/(l+q-ç). By sumption. 1 < r < 2. It then follows Kom the 
Sobolev embedding theorems that 
+ - 
Similarly, we deduce that 
The two previous inequalities ilnply the estinmte 
Using the expression of H given in (Ha), and recalling that I(z) > 0 for large 
we obtain 
(5.z) }   .- -_ 
Next using (5.2), assulnption (Hs) and (5.1), we obtain 
The combination of (5.1) and (5.3) implies 
+ 
Since a < 2 and 2(r - 1) < q. we see that (z,,) is bounded. 
Lemma 5.4. I satisfies (Io). 



STRONGLY INDEFINITE FUNCTIONALS AND MULTIPLE SOLUTIONS 2989 

Pro@ Let (zn) be a (PS); sequeuce wiîh c > 0. Using Lelmlm 5.3, an argulient 
similar to that of Lçlnlna 3.2 shows that along a subsequence zn --" z E/Cc, we bave 
,, -- z in Hot(tl). Since E1 C H01([), we have Pzn  Ptz. [] 

Pwof of Theorem 1.6. Since H(x, z) is evell in z, I is even. Bv assmnption. I(0) = 
0. Lemnlas 5.1, 5.2 and 5.4 show that I satisfies (Is) - (It0)- Now Proposition 2.3 
applies, aiM the proof is COlnplete. [] 

A('KNOWLEDGMENTS 

De Figueiredo was suppolted by CNPq-FAPESP-PIR()NEX. Ding was supported 
by the Special Fuuds fol lkla.iOl" State Basic Research Projects of China. the flmds 
of ('AS/('hilm 119902. lfl01Nl{-I, and the CNPq of Brazil. 

EFERENCES 

1. T. Bartsch, lnfinitely many solutions of a symmetric Dirichlet problem. Nonlinem- Anal. TMA, 
20 (1993), 1205 1216. MR 94g:35093 
2. T. Ba.rtsch and I). G. De Figueiredo, Infinitely mang solutions of nonlinear elliptic systems, 
Progress in Nonlin«ar Differential Equations and Their Applications, vol. 35, Birkh/iuser, 
Basel/Switzerland. 19.q9. pp. 51 67. MR 2000j:35072 
3. V. Benci and P. Rabinowitz. Critical point theorems for mdefinite functonals, lnvent. Math. 
52 (1979), 241 273. 
M1R 80i:58019 
4. K. C. Chang, lnfinite-Dimensional Morse Theory and Multiple Solution Problems, 
Birkh/iuser, Boston, 1993. MR 94e:5023 
5. D. G. Costa and C. A. Maga.lhfies, A variational approach to noncooperative elliptic systems, 
Nonlinear Anal. TMA 25 (1995), 699 ïl5. hlR 96g:35070 
6. D. G. Costa and C. A. Magalh£es. A unified approach fo a class of strongly indefinite func- 
tionals, .J. Differential Equations 125 (1996), 521-547. gin 96m:5N161 
7. D. G. De Figueiredo and C. A. hla.galh£es, On nonquadratic Hamiltonian elliptic systems, 
Adv. Differential Equations 1 (1996), 881-898. Mil 97f:35049 
8. D. G. De Figueiredo and P. L. Felmer. On superquadratic elliptic systems, Trans. Amer. Math. 
Soc. 343 (1994), 97-116. Mil 94g:35072 
9. Y. H. Ding, Infinitely many entire solutions of an elliptic system with symmetry, Topologica.1 
Methods in Nonlinear Anal. 9 1997), 313-323. hlR 99a:35062 
10. P. L. Felrner, Periodic solutions of "superquadratic" Hamiltonian systems, J. Differential 
Equations 102 (1993), 188-207. hlR 94c:58160 
ll. J. Hulshof and Pt. van der Vorst, Differential systems with strongly indefinite variational 
structure, J. Funct. Anal. 114 (1993}, 32-58. hIR 94g:35073 
12. P. H. Rabinowitz, hhnimax Methods in Critical Point Theory with Applications to Differential 
Equations, C. B. M. S. vol. 65, Amer. Math. Soc., Providence, RI. 1986. Mil 87j:58024 
13. E. A. B. Si|va, Nontrivial solutions for noncooperative elliptic systems af resonance, Electronic 
J. Differential Equations 6 (2001}. 267-283. Mil 2001j:3509ï 
14. M. Wiilem. Minimax Theorems, Progress in Nonlinea.r Differential Equations and their Ap- 
plications, vol. 24. Birkh/i,ser. Boston, 1996. hlR 97h:5037 

IMECC-UNICAMP. CAIXA POSTAL 6065, 13083-970 CAMPINAS S.P. BRAZIL 
E-mail address: dj airo@ime, unicamp, br 

INSTITUTE OF ,MkTHEMATICS, AMSS. CHINESE ACADEM5 OF SCIENCES. 100080 BE1JING. PEO- 
PLE'S I:{EPUBLIC OF CHINA 
E-mail address: dingyh¢math03, math. ac. Ch 



TRANSACTIONS OF THE 
AMERICAN MATHEMATICAL SOCIETY 
Volume 355, Number 7, Pages 2991-3008 
S 0002-9947(03)03279-3 
Article electronically published on Match 17, 2003 

STABILITY OF SMALL AMPLITUDE BOUNDARY LAYERS FOR 
MIXED HYPERBOLIC-PARABOLIC SYSTEMS 

F. ROUSSET 

ABSTRACT. We consider an initial boundary value problem for a symmetriz- 
able mixed hyperbolic-parabolic system of conservation iaws with a smail vis- 
cosity e, u[ + F(u«)x = e( B(u «)uex)x. When the boundary is noncharacteristic 
for both the viscous and the inviscid system, and the boundary condition dissi- 
pative, we show that u « converges to a solution of t he inviscid system before the 
formation of shocks if the amplitude of the boundary iayer is suflïciently smaii. 
This generalizes previous results obtailmd for B invertible and the lincar study 
of Serre and Zumbrun obtained for a pure Dirichlet's boundary condition. 

1. INTRODUCTION 
We consider a one-dimensional systenl of conservation laws with a small param- 
eter « set in the dornain x > 0. 
(1) u t + F(uS): = e(B(uS)v)x, x > O. t > O. 
where u e G n and F • H  n, B • H  nxn. ç will assume that F and B 
are smooth (Cm). We add to this system an initial condition us(0, x) = u0(x) and 
a boundary condition that we will detail later.  sume that the eigenvalues of 
B have nonnegative real part and that the tank of B does not depend on u. Ve 
will denote it by r, 1  r  n. Note that B is not necessarily invertible. We are 
interested in the limit of u e when « tends to zero. çX expect that u e tends to a 
solution of the inviscid problem: 
(2) ., + F() = 0 
with sonle bomldary conditions to be determined. At first we make the natural 
sunlptions to ensure the well-posedness of the Cauchy problem for (1) [6]. There 
exists a change of variable u  v(u) with inverse u = 9(v) in which the systenl can 
be rewritten as 
(3) g(v)t + f(v) = e(b(v)Vx) 
with the following properties: 
• (H1) b(v) is block diagonal, 
b(v) = 0 bi(v) " 
with b (v) e GL(). 

Received by the editors January 30, 2002 and, in revised form, December 13, 2002. 
2000 Matheraatics Subject Classification. 35K50, 35L50, 35L65, 76H20. 

()2003 American Mathematical Society 



2992 F. ROUSSET 

• (H2) dg(v) is lower block diagonal. 

dg(v)= ( v O) 
(,) " 

with t)(v) E GL,-(). 
Bv analogy to the terminology in gas dvnamics, we shall refer to t' as the primitive 
variable. 
Next we assmne that (1) is symmetrizable mixed hypert»olic-parabolic: 
• (H3) there exists a positive definite symmetric (u) such that 
(1) E(u)dF(u) is symmetric, 
(oe) (,)(.)x. x  l()Xl =, x  , 
where  > 0. mM - stmMs for the scalar product of . 
k2, denote by v = (w, z) the corresponding block deeomposition of ç. 
Note that since [(v) =dF(u)dg(v), b( v)= B(u)dg(), setting S()=dg(v)t(v), 
we get that (H3) is eqnivalent to 
• (H3') There exists S(v) such thm, 
(1) S(ç)dg(v) is positive definite svmmetric, 
(2) S(v)df(v) is svmmetric, 
(3) S(,)b(,)x. x > .ll , vx = e 
$$% point out that (tt3)(1) implies that the inviscid svstem (2) is hyperbolic. 3Io» 
over (H3)(2) in,plies that dgtd9 is i,h,ck diagonal (see [11], Lemma. 4.1). Hence 
thanks to (H2), ce get 
• S(v) 
where S(v) is positive definite symlnetric. Consequently, writing the block de- 
composition of df as 

we get flore (H3')(2) that Sw(v)h(v) is symmetric. This means that the system 
obtained from (3) by removing the second equation is symmetric-hyperbolic. 
Finallv. we also assume that the hyperbolic and parabolic modes do couple: 
• (H4) The kernel of B does hot contain anv eigenvector of dF. 
The structural hypotheses (H1-H4) are verified by many physical equations as 
those of compressible gas dvnamics and nmgnetohydrodynamics. 
Next we make hypotheses to deal with the initial boundary vahle problem. We 
focus on the case of a noncharacteristic boundarv. We assmne that the boundarv 
is noncharacteristic for both the viscous (1) and the inviscid (2) systems: 
• (H5) dF(u) and ](ç) are nollsingular. 
Note that an inflow or outflow boundary condition nmkes the boundary noncharac- 
teristic in most cases for the Euler and Navier-Stokes equations. These boundarv 
conditions have a physical meaning since they appear iii problelns with aperture. 
such as in oil recoverv. The analysis of an impermeable boundary xvould be different 
since in this case the boundary is characteristic. 
We denote by q the number of eigenvalues of positive rem part of dF(U), and 
by p the nmnber of eigenvalues of positive real part of h(v). An initial boundary 



STABILITY OF BOUNDARY LAh ERS 2993 

value problem for (1) nee(ls p + r scalar indepen(lent bomdary conditions, and 
an ifitial boundary lue problem for (2) needs q independent scalar boundary 
conditions. We deal with bounda.ry conditions fl»r (3) that are linear with respect 
to the primitive variable v. çV write the bOulidary condition for (3) as 
(5) Lv(t'O)=(l'") (''O)=9.z 
where l is a linear mal» that has rank p and 9 is a given constant. 
In the following, in order to make energy estinlates, we assmne that the boundary 
condition (5) is "dissipative": 
*(H6) fl > 0. Vv  ç, VX = .¥ . such that Lv = 9- LA = 0. and wc 
have 
s(v)(v)x . x  -lXl . 
There are physical boundary conditions in the form (5) that satisfv (ti6). The 
case of the isentropic gas dvnamics will bc stud;ed Iwh»w. 
Note that thanks to hyl,otheses (H I-Ha), we have p ç q  p + r ([12]. Corollary 
1). Hence in the case q < p + r. there is a h»ss of bOmldarv condition when  tends 
to zero. It is due to a fast change of u  in a vicinitv of the boundarv: the b«nmdarv 
laver. In the noncharacteristic cas« the size of tlw bomdary laver is ¢. When ¢ 
tends to zero. the expected t,ehaviour of ,.« is ([2], [1], [12]) 
  ,.it(t. z) + U(t..r 

where u it is a soluti(,n of (2) with the initial condition ,t(0..) = u0(z) and some 
boundary conditions that we have to deternfine. U(t, z) is a boundary layer; it is a 
solution of a differential problem where the time is onlv a parameter: 
(6) (U)U' = (U + ,,'(t. 0))- F("(t.O)). 
U(t, +OE) = o. 
L,(U(t.O) + ,,"'(t. 0)) = g. 
Note that when r < , we have an algebraic differential system. This problem has 
solutions if and onlv if uit(t. 0) belongs to the subset C. where 
c = u+  ", u, u(+oc) = o. 
L,(U(O)+r+)=g. 
This sct C is called the set of residual boundary conditions. It was studied in the 
case r = n in [2], [4] and ill the gelleral Ça,S(? iii [11]. Assuming that u0(0) satisfies 
the boundary condition (5), i.e., 
(7) L(o(0)) = 0. 
we have 0(0) C (the associated profile ofthe boundary layer is U = 0). Moreover, 
thanks to (H1-H6) we can use [11], Lemma 4.2 and Theorem 1.1. C is a smooth 
submanifold in the vicinity of u0(0) that bas dimension q and that is transverse 
to the unstable subspace of dF(u0(0)). Consequentl3; thanks to a theorem of [8], 
there exist.s a continuous solution of (2) with the boundary condition uit(t, O)  C 
defined on [0, T] for some positive small time T. Assunfing some higher-order 
compatibilities between "u0(0) and C, we can even get a smooth solution u. Usiug 



2994 F 1ROUSSET 

the smne method as in [4], we can show the existence of an approximate solution 
of (1) in the fornl 

(8) 

such that 

where 

^1 
i=1 

Lv(u«vP(t.O)) = O. uaPv(O.x) = uo(C). 
0,  + OF(, ) - eOx(B()Ox) = R  

]]R¢E]IL[O,TI,L 2 __ Cg M. 
Our aire is to show that the truc solution a f is close to the approximate solution if 
the boundary laver is sufficiently weak. More precisely, 
Theorem 1 (Nonlinear stability). Assumin9 (H1-H6) and that uo 6 Hr (+ ), there 
eists  > 0 such that if 
(9) sup (lO:U(t,')l + zlO:U(t,z)ldz ]O=U(t,z)ldz  a, 
tG[0.T] 
then 

tf e -- ri int --* 0 

(o) 
with 
() 

((()«f(v))  ÷ S()b(@X. X _ OlXl , 

VX  IR ' 

k(v)dg(v) skew-synmmtric. 

when ¢ -- 0 in L([O,T],L2). 
To prove this theorem, we actually need to start from a very accurate approx- 
imate solution -u app. Indeed, we will take M = 3 in the expansion (8). The con- 
struction of such a high-order expansion requires a lot of regularity on u i't (sec 
[4]). This is why we have to assume so much regularity on u0.  actually get a 
more precise estinmte: 
I1(. « -- .«)(t, ")11= + «110=( « -- )(t, ")11 + «11( "« -- )(t, .) Il  w,.  C«- 
Our method can also provide estinmtes in L([0, T], H ) for anv s. 
The proof of Theorem 1 relies on energy estimates. We use the primitive variable 
v; hence we work on the form (3) of the equation. X combine the energy estimate 
of the totally parabolic case (r = n) [2], [4] with an energy estimate of Kawhima's 
type [6] and a careful study of the boundary values. A key argument of the proof 
is the following lemma of [la]: 
Lemma 2 (S-K [13]). Ass.uming (H1-H4), there eœists a skew-sBmmetric Iç(.u) and 
a positive constant 0 such that 
((Iç(u)dF(u)) Æ + (u)B(u))X. X  OlXl =, vx  , v..  u 
'(KdF+(K«F) t) 
where (KdF) Æ =  . 
Note t, hat setting k(v) = dg(v) tK(9(v)) we ean rewrite this result  



STABILITY OF BOUNDARY LAYERS 2995 

Iu [9], Lenmm 2 aud the estimates of [6] combiued with pointwise Green's lune- 
tions bouuds were Mready used to prove the nonlinear asymptotic stability of weak 
time-iudependent viscous shock profiles for (1). The asylnptotic stability of a time- 
indepeude,t profile of the boundary laver together with the stability of other non- 
line waves was studied in [10] for the isentropic gas dvnamics rewritteu as a p 
system in Lagrangiau coordiuates. 
Let us give an example of an application of our theorem. C, ousider the isentropic 
gas dvuamics where v : (p, v), p being he mass deusity and v the fluid velocity, 
pv ' pv  + p(p) ' o (p) " 
Here we assume that u > 0 and that p' > 0 (hyperbolicity). The sound speed is 
c(p) = . (H1-H4) are verified: moreover, the eigenvalues of dF are v • c and 
the eigenvalue of (the 1 x 1 matrix) h is v. 
Let us first consider an outflow bouudarv condition 

(12) 
(13) 
(14) 
where 

(t. 0) = v- 
with v- < 0. In this ca, l = 0, mM (H6) I)ccomes 
v-}Xl 2 <-lXI2; 
hence, it is satisfied. The compatibility condition (7) becomes vo(0) = v_. It 
suffices to impose Vo((I) + c(po(0))  0 to get (H5). 
If we consider an outflow boundary condition 
v(t,0)=v_, p(t,0)= 
where v_ > 0, we bave 1 = Id, b:er i = {0}, and hence (H6) is true. The com- 
patibility condition bccomes po(0) = p-, vo(0) = v_ and hence we get (HT) if 
v_ -c(p_)  O. Moreover, in the case v- -c(p_) > 0, we have q = p+r = 2; hence 
there is no boundary layer and the hypothesis (9) is always satisfied. 
For a more general discussion of the various boundary conditions for the non- 
isentropic gas dynamics, we refer to [12]. 
As in the totally parabolic case r = n, the smalluess assumption (9) in Theorem 
1 is linked with the stability of the bomMary laver. In [12] an example of a large 
unstable boundary la.ver is given. To understand the mechanism of instability in 
the boundarv laver, we set 0 = 
 . , z = , we fix some time T in app and we linearize 
about the leading terre of u app with respect to . We get the linear svstem 
Oou = 
Ldv(u(r, O) + U(T, 0))v(0, 0) = 0, 
(0, z) = o(z) 

£-u = (b(ut(T,O) + U(T,Z))U' + dB(ut(T,O) + U(T,Z))uU'(T,Z) 
- d((,0) + U(-,z)) . 
Here ' stands for 7" We will say that the profile of the boundary layer u it (v, O) + 
U('r, z) for some fixed "r is linearly stable if t.he solutions of this system tend to zero 



2996 F ROUSSET 

when t tends to +o« The linem" stability is linked with the spectral stability as 
was show,, in [15], [9]. [141. Let us define the domain of/2,- as 

/9(£-)= u=dg(l/(T,z))v, v= . w e HI(]+), z e H2(]+), Lv(t.0)=0 
z 

where V(r, z)is defined by g(V(T,Z))=iWt(T.O)+ U(r, z). 
In [12], it is shown that the essential spectrum of /2- is confined in {7.eA < 
(I} U {0} thanks t.o (H1-H5). In the unstable half-plane {7.eA _> 0}\{0} the spec- 
trum only consists of eigenvalues. Conseqn('ntly, a necessary condition for the linear 
stabilitv of the boundary laycr is that the operator/2- does hot have eigenvalues in 
the unstable half-plalm {7.e A >_ 0}(spectral stability). An Evmls flmction machin- 
ery was developed in [12] to find sufficient conditions of instability. 
In the first part, we show that spectral st.ai)ility holds for weak boundarv lavers. 

Theoreln 3 (Spectral stability). There exists 5 > 0 such that, assuming (HI-H6) 

(15) 

thon F_«. docs hot bave ei9cvalues i the unstable hall-plane {R.c A _> 0}. 

The proof also relies on energy estimates. We first give a direct proof of Theorem 
3 because it seems more enlightening to present the main ingredients of the proof 
of Theorem 1 in the simpler linear time-independent setting of Theorem 3. This 
result is not used in the proof of Theorem 1. The result of Theorem 3 could be 
deduced from direct energy estimates on the rime evolutionarv svtenl (12), (13), 
(14). Nevertheless it is interesting to study the spectral stability since ve can expect 
that, as in the totally parabolic case, the sharp assmnption of spectral stability 
implies the nonlinear convergence result [5]. 
Note that out result of Theoreln 3 (obtained bv a different method) implies 
the result of the appendix of [12] where only Dirichlet's boundary conditions were 
considered for (1). 
In the second part, we give the proof of the full nonlinear stabilitv result of 
Theoreln 1. 

2. SPECTRAL STABILITY 
In this section, we prove Theorem 3. We studv the eigenvalue problem 
(16) Au- £«u = O. 
(17) Ldv(uit(T.O) + U(r, 0))u(0) = 0. 
Setting Ut(T,O)+ U(T,Z) = g(I'),  = dg(I')v (ve onfit the dependence with 
respect to r in this section since r is fixed), we rewrite the probleln in the prinfitive 
variable. Hence we have to study the equation 
(18) AA°v + Ae' - (bv')' = A'v + (Cv)', 
(19) Lv(0) = (lw(0) ) 
:(0) =0 



STAI3ILITY OF BOUNDAR LAYERS 2997 
where v = , An(z) = dg(I'), A(z) = df(l'), b(z) = b(l'), and Ch 
z 
db(V)hI a. Note that we have the estimates 
for some M > O. where S(z) stands tBr 
Moreover, note that thauks to (H 1), 
(21) II(Cv)' = O. 
where Hw = w. 
Z 
Let us assmne that there exists a nonzero soluti.n of (18), (19); withom 1.ss of 
generality, we assume that 
(22) IIvll = 1. 
In this section, since we dcal with flmctions that take complex values, we denote 
bv u • v the scalar product of C n, 
and by - the associated uorm.  th«u define 
II,ll 2 = -0., (.) = ,«()-,,(.)dz. 
 split the proof of the theorem into several lemmas. We will collect, all the 
estimates at the end of the section to reach our conclusiou. 
 first give an energy estimate iu the saine spirit as in the totally parabolic 
case [2], [4] or in the pure Dirichlet's bouudary coudition case [12]: 
Lemma 4. Assm,e that v is a solution 4 (18). (19) that satisfies (22). Then. 
when  is scieztlg small, we have the estimate 
(z3) n  + l'll  + l,(0)l  5 c. . 
(24) «  + 11'112 + lw(0)l =  cs(I,.(0)l = + I1'11=), 
(25) IZ, 1  c( + I1"11). 
Note th.at the first estiçmte (23) 9ives 
(26) e   C3. 
Proof.  first use the saine energy estimate as in the strict]v parabolic cse [2], 
[4] and the full Dirichlet case [12].  take the Hermitian product of (16) bv 
(in this section, we wil] denote S(V) by S for the sake of simplicity) and we take 
the real part, getting 
« (SA%, v) + e (SAc', v) - « (S(bv')', v) = e (SA'v. v) + Re (S(Cr)', v). 



2998 F. ROUSSET 

Since SA is syminetric, we get 
Tae (SA-d, v) = 

> 
thanks to (tt6). Note that 
I(SA)'v,v)l <_ C + 
Next, integrating by parts, we bave 

1SAv(O).v(O) 
 ( (SA)'v, ,,) - - 
-((SA)'., ,) q- cl,(o)l 2 

IV'llvl 2 

Tac ( s(bv)', v) = -R « ( Sbd, ,') - n« ( ( S'bv', ,) - Tac Sb,' (0). v(O). 

Thanks to (H3'), we have 

nloreover, we bave 

« (Sb',') _> zlz'12; 

Tae Sbd(O) . v(O) = 0 

thanks to the structure of the matrix b given by (H1) and (19). Using again (ttl) 
and (4), we have 
( 1 0+° ) 
(('bd,.)) = (<b,z',z) < C ollz'll 2 +  IV'llzl 2 
for every q > 0 by using the Young inequality. Moreover, we have 
[(A'v, v)l _< C IV'I Ivl 2 dz 

and 

(27) I(S(o)',,)l = 
J0 +oe lJ0+°c ) 
< c IV'l MIz'l < C@ll'll 2 + IV'lll  
thanks to (H1) and (20). 
Çollecting these various inequalities, we have shown 
«ll.vl12+llz'l12+l,(o)l 2  Cllz'll2+C() IV'llzl2+C() IV'lM 2. 
To conclude, we first choose  = , then we use z(0) = 0 through the inequality 
14x)l 
+ c[ + _ 
(28) IV'lll  < xl"lll»'ll  < Cllz'll  
and finalb; we absorb the terres Celle'Il  l,d Ç(e) f IV'III  i, th i prt 
11'11  f   -me,tb" n. hi proe (23). 
To get (24), we use 
(29) 
IV'll,l   olV'lll'll  + I(O)1  IV'l  O(l(o)l  + II,'ll). 



STABILITY OF BOUNDARY LA' ERS 2999 

To prove (25), we also take the scalar product of (16) by St,, we take the imagi- 
uary part and we onlv use 
z,,, (s,,.", ) : -z ( Sb,', ') - Z,,, ( ( Sb)',,', ,,)  C(llz'll  + I1'111111)- 
 get 
z, llll  5 C(ll,'ll  + I1'111111 + allll) • 
To conchlde, it suites to use (23), whi«h gives, in pa.rticular, 
I1'11   Call,,ll  
and the normalization assmnption (22). 
In the case of a pure Diri«hlet boundary condition, a weighted energy estimate 
on the hyperbolic part of the system (that is to say on the first n - r equations) 
w used in [12] to bound the terre 
This estinmtç was similar to the one used by (ioodman [3] for tire stabilitv of viscolls 
shock profiles. This was ecient because of thc upwind propagation. In out more 
general seting we use an energy estimato of "Kawashima's typo" [6], [9]. 
Lennna g. Assume lha¢ v is a solvtion of (18), (1.9) tat sat.sfies (22). Ten for 
scieal small . we have 
(30) II.'ll 
Proof. We use the nmtrix k givcn by (10).  apply b to (16), we take the scalar 
product by v' and we take the real part. Using "e(kAv', v') = ((kA)*v ', v'), we get 
Here we have used the estimates (28) and (29). 
Using that kA ° is skew-Hernlitian, we have (kA°t ', v') E R since 
(a-A°v, v') : kA°v(O), v(O) - ((A°)'v, v) - (kA°, ', v) = -(A".'. v). 
Consequently, we have 
I«(a(A%, ¢))1 : Id )(.4 , ¢)1 S C«allll I1¢11. 
Siuce we have the estimate (10) 
((A)''. ')  011'11  - ClI'II , 

we get 

IIv'll 2 < C(llz'll 2 + IIz"ll IIv'll + ne  I11111'11 + all'll 2 + al'(o)12). 
and hence choosing r/> 0 sufficiently small, using the Young inequality and (9) we 
bave 

/ 
(31) II'll 2 < c(,1)l, llz'll 2 
+ 
Consequently (30) is proved. 

(ne x)211,112 +- I1»"112 + al,(o)12). 



3000 F ttOUSSET 

To end the proof of the theorem we would want to estimate []z"[[ with respect 
Lemma 6..4.ssne that  is  solution of (18), (19) that sati.sfies (22). Then. 
when 6 i,s stcientl sntall , u,e bave 
I1"11 
(32) 
Pçoof.  take the derivative of (16), getting the equation 
",' + A,," - (b,'")' = O(IV'l)(l I + I"1 + I'1) + (C)". 
Thc proo[ is very similar to the proof of (23), in that we take the scalar product of 
t.he equati(m hy Sv  and we do an integration by parts. The «boundary'" terres do 
hot vanish since d(0) does uot satis" the bOulMary condition (19). b just point 
mt that to lmund the terln ((S(Cv)",v') we also do 11 integratiou bv parts as iu 
(27) to get an estilnate independ¢,lt of 
2.1. Proof of Theorem 3. ke now give the proof of Theorem 3. To conclude. 
we first have to eliminate z"(0) and v(0) in (32). 
We first express w'(O), lhanks to thc hyperboli« part of equation (18): 
,.'+ .a, ,' + .a.=' = O(1"1)(1 I + I'l). 
where A = A. .4 . Note that we make a crucial use of (21). 
Since the boundarv is non«hara«teristic for the viscous svstem. A is nonsingular: 
moreover, thanks to (2g), (25), we have 

(33) 
We dedu«e 

I,'(o)12 _< c'(l,(o)l 2 + I»'(o)12 + 1,(o)1211v'll 2) 
(34) <_ C(l,,(0)l 2 + I-'(0)12 + llv'll 2) 
since thanks to (2g). we have w(0)]   
The next step is to estimate lz'(O). We use the classical Sobolev inequality 
(35) Iz'(o)l oe  211»"1111zll  ,#llz"ll oe + llzll oe 
for every q ) 0. Hence it SllCeS to estimate z"(0) in (32). %7 use the parabolic 
part of the equation 
4 ' 
O(v) + Aa,«' +. : -b," = O(IV'I)(II + Idl)+ 0(1111). 
We get. thanks to (23), (25), (34), and (35), 
1 12 11,,112) 
(o) I"(o)12  c'(l(o)l 2 + ( + ,])11'1 + 11'112 + 
Next. we choose q such that C'q < 1, and we replace (34). (35), (36) in (39]. getting 
(7) I1"112  ç(lldll 2 + I1'112 + 1,,(o)12). 
çinally. «oll«ting (23), (30) and (37), w bave shown that 
(« )( - raz) + ( - c')ll«ll  + ( - C)l.,,.(o)l 2  o. 



STABILITY OF BOUNDAIY LAYEIRS 3001 

Hence if 5 is sufficiently small, this gives if T¢e A > 0, z = 0 and w(0) = 0. The 
hyperbolic part of the equati(m then t)e«omes a first-older ordinarv difl,rential 
equation involving only w: 
,,' : (A)-(-X, ' + 0(1"1),') 
with the boundmT condition w(O) = 0. Conseuently we a.lso get w : 0. This ends 
the proof of Theorem 3. 

3. NONLINEAR STABILITh 
In this section we prove Theorem 1. We use the form (3) of the svstcm. Setting 
u e = g(v e) and u app = g(vaPP), we have the two systcms 
J(Ve)t ÷ (f(ve)x = 
Lve(t. 0) = g. 
,'e(o. *) = ,'o(.r) 

and 

Setting v E = v app + v (we Olllit thc dependence of t, iii ), We rcwritc Olll" lrollt'lll 
s 
A°i)tv + Ac%v - eO.(bO:v) = R G + 310 + ]111 + 

(38) 
where 

A ° 
b = 
M ° = 

AI 1 = 
11 '2 = 
Note that v satisfies the 
(39) 
and the initial condition 
(40) 

L,(t.O) =0 

(41) 
where 

v(O. x) = O. 
We «hoose C suffi«iently large such that 
Q <_ Ce N, 



3002 F. ROIJSSET 

for solne large N which will be chosen later. 
To plove Theorem 1. we lise the classical çontinuous induction argmnent ([3], 
[7], [51, [9]). Let IlS define 
+ «Oz(s)l[ = + «ll0,z(s) = d. 
+ t,(s. 0)l = + «=10¢,(.. 0)1 = + «410,¢,,(t. 0)1= d.. 
Note that thanks to (39), (40), (4), we have 
(42) E(o)  ç'e . 
Using the classical short-lime theory, we define 
r* = ,,p{ç« e [0. ç],   oh,ti,,, of (3S), (39), (a0) 
e [0. r). E(t)  e ' } 
Sll(.h 
that 
Vt 
where we choose Ni < N. There are wo possibilities: 
() T* = T, 
(2) T* < T. and E(T*) = e N. 
In the following, we show bv an energy estimate that we cannot be in the second 
case. This will show Theoreln 1. 
Let us define a()  
Z = Slip slip 00îI" t,-- . 
a ê te[0.T] 2>a>l,2kk0' 
Al first we need an elementary lemlna about the estilnates of the nonlinear quan- 

tities that arise in (38). 
Lemlna 7. Vi = 0.1. 

IOAI < c(10«vl)(1 -4-10,vl), 



STABILITY OF BOUNDARY LAYERS 3003 

/ 
Remark 8. 
  '--t x '* Z 
 also point out that, actually, we will hot usç the case o = 2,/3 = l in Lelnlna 
7. 
ç now corne to tlw proof of oto nmin theorem l. In the proof. C stands for a 
number that is independent of ¢ but nmy dOl»end on T. 
Since bv classical Sobolov emtmddings, we have 
T* (T* 
1Ozlœe 
and since by using the hyperbolic part o[ efluation (38) (thal is o say, the w 
component), the nonchmacteristic assmnption and Remark 8. we bave 
we gel the estimale 
(43) 
Ce,- 
Çonsequcntly, we choose N  6 and 5 < N < N. This allows us to use Lemma 7. 
Moreover, thanks to (43), we oi)tain })y contimfity ri'oto (H6) that 
(44) S,,,(v 
since 
Note that our smalhmss assmnption (9) and (6) imply that 
(45) p (z)+ () d: + () dz 5 C. 
z6N+ , , 
As for the spectral stability, there are four steps in the proof. Ve first lnake 
the energy estilnate of the totally parabolic case; next we lnake an estimate of 
Kawashima's type and an estimate on the space derivative of equation (38). The 
final step is to estimate the boundary values. For this, we replace (25) in the time 
evolutionary setting bv ai1 energy estilnate on the time derivative of the equation. 
At first, let us make the saine energy estilnate as for the totallv parabolic case 
[2], [4]. 
- 

(46) 
,0 ÷°c 

a(ç)lvll0xvl _< CllOvll a(-)lvl 
_< c(«ll0.ll 
we easily get., after absorbing the terres CS]w(t. 0)12 by the ter,,, ct[w(t, 0)12 
left-hand side, 
(48) Ot(SA°v,v)+a[u'(t,O)[ 2 
where S stands for S(1 app ÷ ,). 

on the 



3004 F. ROUSSET 

Note tlmt an estinm.te such as (47) is needed to bound the teInls 
(eSi:).(bOxv), z,), 
Next we replace the estinmte (25) of the spectral sta.bility bv an estinmte on the 
time derivative Ov. Since 0«¢, still satisfies the bomdary condition LO«v = 0. we 
can perform ¢he saine computation as previously on the rime derivative of (38). 
Thanks to Lemnm 7. we get 
(4) o,(sA%.o,)+lO,,,(t.o)l=+5dlO,.ll = 
the ideas of the computation have been use& 

We do hot give more details since ail 
\Ve just point out that we bave used 

(50) 
and that fo bound the terre 

we perform an integration by parts and use the block assunlption (H1) and (4). 
This ternl is then dominated bv 
C(ll,ll = ÷ 11,=1 + qllo,=ll =) 
for every q > O. , absorb the last factor by the terre flllO,.=ll  . the left-hand 
side v choosing 1 suciently small. 
Note that for the moment, we do hot use an ineqlmlity similar to (46) to bound 
terlns Sllch as 
+oe a()[tt' 2 
 bomd this terre by expressing Ott', thanks to equation (38) and by using esti- 
mates such as (46) and (47). Then we get 

COIlleS frolll 

(Ç ' ) 
_< c I,.(¢.0)1 =÷-IIôx¢,ll = . 



STABILITY OF BOUNDARY LAh ERS 3005 

Replacing (51) in (49) yields 
(5"2.) ,(S'A",,,.O,,,) ÷ lO,.,(t.o)l 
I 
 c( « + I111 = + Ila,,ll = + (] + )lla=l +  • 
As for the spectra} stability, thç next step is to use the "Kawashima'" estimate 
[6], [9]. b aI,ply k(v 
(53) (kA°0,,..O,v) +(('- "' . - 
 use the cruciM estimate (10). This 

Next we xvrite for every q > O, 

= I«(»O,.bO,.,..O,v) + 
< ('((« + )110.,'112 + llO,zlli.,,ll) 
C((e + d + )110,11 = + 
where here we bave used the blo«k structure assmnt)Iion (l[ l) and tlw 5omig in- 
equality. Using Lemma 7, the ung inequality, and (47) we bave 
(55) 
Bv the smne method, we ge simi}ar estimates for (kM °, O.u) and (kM 2. Ozt'). To 
handle (kA°&u. 0«v), we write 
Performing an integration bv parts in the lt factor al»ove, we get 
+ (¢(kA°)t,.Otv) + 
and hence, since kA ° is skew-symmetric, 
1 
(O,(kA°v.Ov) - (Ot(k.4°)v.O«v) 
(kA°Otv. Ov) = 
+ kA°(t. 0),9.0). O,,,(t. O) + (O(.A°),,, 0,.,.)). 
To bound (O«(kA°)v. O,v). we uoe 

- ( )MIO, vl d« 

thanks to (46) and (51). 



3006 F ROUSSET 

This yields 
1 o 
(56) (kA°Otv. O«v)=- 
for any q > 0 by use of the ung inequality. Consequently. collecting 
(56), choosing q sufficiently small and using our assumption (9) to absorb (e + 
)ll0vll  in the left-hand side if  is sufficiently small, we get from (53), 
+ 
In conchlsion, it relnains to estilnate 
with respect to z and we perform an energy estimate similar to (2), but now the 
boundary terres in the integration by parts do hOt vanish. Using Lemma 7 and 
estimates such as (46), (47), and (50), we get 
(58) O,(SA"Ov,Ov) + llOll  
 c  + (1 + 
+ Iou(¢. 0)1  + 10(, 0)1). 
Note that to estimate the terre (Ox12,âxv) we have perfomed an integration 
1)y parts to avoid that terres involving IlOfll appear and that to estimate 
(SOAaOcv, Ov), we write 
f+ 
I(SOA°O,,Oæ)I 
and we use (51). 
As for the spectral stability, the next step is to estimate the boundarv values 
Ov(t,O) a,,d 0(.0).  rt write th ,og, of (aS), 
1 
(59) 1o.(t. 0)l 2 
for some q sufficiently smal] so tiret the terre 
wi]l be absorbed by g]e ]efg-hand side of (58). 
To est.image Ow(t, 0), we use the hyperbolic parg of the equation, the facg that 
the boundary is noncharacteristic and Lemma 7, getting the estimage 
IOw(t,0)l   c(IR«(t,0)ff 
(60)  c(l(t, 0)1 
1 



STABILIT OF BOUNDAR3 LA5 ERS 3007 

To estimate O.zz(l, 0), we use the parabolic part of the equatioll. This vields 
(6a) 2lo=z(t.o)12  C(l(t.o)l 2+lo,.,(t,o)l =+«,llo.=zll  
+ --0»1  + ( +  , - 
Next, putting (59), (60), (61) in (58) gives the bound 
(62) Ot(SA°Ov,Oz ,) + «]]O=zll 2 
(    ,,,,,, 
 c 0  + (a + )1111 = + (a + )110.11 = +  . 
Fina.lly, we «onsider (48) + «2(52) + «(57) + F«2(62) with F > 0 suffi«ientlv large 
and independent of . lntegrating ri'oto 0 to t, we get the estimate 
(63) IIz,(t)ll 2 + 2110z'(t)ll2 + 2ll0=(t)ll 2 
+ Ç «ll0z,(s)ll 2 + «31lOtz(s)ll 2 + l'«3lloz(s)ll 2 d.) 
 ç IIz,()ll 2 + «2llO,,,()ll 2 d. + e N) 
thanks t.o (dl), (2), and (9). Note that we havc also used that N'.4" is positive 
definitc symmetri«. Thanks to 5ung's inequality, we bave 
for any q > O. Hen«e «hoosing q su«iently snmll su«h that Çq < 1 and then F 
C 
sucient}v large su«h that ç > , we finallv get 

«allO, xvll = + alO,.,(t, o)1 =. 
Next using Lemma 7. we perform estimates analogous to (49), (57), and (62} for 
Ottv, Ot«v and Ot«,z respeetively. 
Finally. we obtain the estimat.e 
E(t) <_ c(N + fotE(s)d.s). 
Therefore, by Gronwall's Lemma we bave 
:(t) < c«  vt  [o.w*]. 



3008 F. ROUSSET 

Hence E(T*) < ¢N since N > Ni and hence T* = T. 

1 EFER.ENCES 
[1] M. Gisclon, Étude des conditions aux limites pour un système strictement hyperbohque, 
via l'approximation parabolique, J. Math. Pures Appl. (9) 75 (1996), no. 5, 485-508. 
97f:35129 
[2] M. Gisclon and D. Serre, Étude des conditions aux limites pour un système strictement 
byberbolique via l'approxinatio parabolique, C. R. Acad. Sci. Paris Sér. 1 Math. 319 (1994), 
no. 4, 377-38-'2. NIR 95e:35119 
[3] J. Goodman. Nonlinear asgmptotic stabilitg of viscous shock profiles for conservation laws, 
Arch. RationM hlcch. Anal. 95 (1986), uo. 4. 325-344. hlR 88b:35127 
[4] E. Grenier and O. Guès, Boundary layers for viscous perturbations of noncbaracteristic 
quasilinear hgperbolic problerns, .1. Differential Equations 143 (1998), no. 1, 110 146. MR 
98j:35026 
[51 E. Grenier and F. Rousset. Stabil,tg of one-dimensional boundary layers by using Green's 
fttncttoï)s, ('.ondin. Pure Appl. Malh. 54 (2ri01), uo. 11. 1343-1385. MR 2003a:35126 
[6] S. lçawashilua. Sgstcms of a hgperbolic parabolic type witb applications to the equations of 
magetohydrodgamtc«, Ph.D. lhesis. Kyoto University (193). 
[7] G. Kreiss and H.-(ï). Kreiss, Stablity of systems of viscous conservatzon laws, Comln. Pure 
Appl. Math. 51 (1.q98), no. 11-12, 1397 1424. hlR 2000c:35156 
[8] T. T. Li and W. C. Yu. Boundary value problems for quastlincar hgperbolic systems. Duke 
University Malhematics l)epartlneut. Durham, N.C.. 1985. NIR 88g:35115 
[9] C. Mascia a.nd N. Zumbrun. Stçbility of wscous shock profiles for dissipative symmetric 
hyperbolic-parabolic systers, Preprint (201). 
[10] A. lkIatsumura and K. Nishihara, La,ye-time behaviors of sobdions to an znflow problem 
in tbe balf pace for a one-dinensional system compressible viscous gas, Commun. Math. 
Physics 222 (2001), 149 474. MR 2002rn:76083 
[11] F. Rousset, The boundarg conditions coming from the real vanishmg viscosity method, Dis- 
crete Continuous Dynamical Systelns (to appear). 
[12] lï). Serre and K. Zumbrun. Boundary lager stability in real vanishing viscosity limit, Commun. 
Math. Phys. 221 (20[)1), 267-292. 
[13] Y. Shizuta and S. Kawashima, Sgstems of equations of hyperbolic-parabolic type with appli- 
cations to the discretc Boltzm.ann equation, ttokkaido Math. J. 14 (1985), no. 2. 249-275. 
h'lR 86k:35107 
[14] K. Zulnbrun. Multidimensional stability of planar viscous shock waves., Advances in the 
lheory of shock waves, Prog. Nonlilmar Differential Èquations Appl. 47. Birkhaser, Boston. 
MA. 307 516 (2001). (Ènglish}. hlR 2002k:35200 
[15] K. Zmnbrun and P. Howard. Poitwise semigroup methods and stability of viscous shock 
waves, Indiana Unir. hlath. J. -17 (1998), no. 3, 741 871. NIR 99m:35157 

ENS LYON. UhlPA (UMR 5669 CNRS). 46, ALLÉE DITALIE. 69364 L",ON CEDEX 0ï. FR.-kNÇE 
E-mail address: frousseteumpa.ens-lyon, fr 
Curret address: Laboratoire lï)ieudonné, Université de Nice-Sophia Antipolis. Parc Valrose, 
06108 Nice Cdex 02, France 
E-mail address: froussetemath.unice, fr 



Editorial hlformation 
To be published in the Transactions, a paper must be correct, new, nontrivial. 
and significant. Further, it must t)e vell writtcn and of interest to a substantial 
numher of mathematicians. Piecemeal results, such as an inconclusive stop toward 
an unproved major theorem or a minor variation on a known rcsult, arê in general 
hot acceptable for publication. 
Papers submitted to the 7ra.sactions should exceed 10 published journal pages 
in length. Shorter papers mav be sul)mittêd to the Proceedings of the .4merican 
ll[atheroatical Societ.q. Published pages are the saine size as those generated in the 
style files provided for OE,v/$-I35FX or OE/,_ç-l\. 
As of Fel)ruary 28, 2003, the backlog for this j«mrnal was approximately 2 issues. 
This estimate is the result of dividing the number of manuscripts for this journal 
in the Providence office that have hot 3"et gone to the printer on the above date bv 
the average number of m'ti«les per issue over thc previous twelve months, reduced 
1)y the numl)er of issues l)ublishcd in ff)m" months (the time ne«essary for editing 
and comt)osing a typical issue). In an eN»rt to make articles available as quiçkly as 
possible, articles are posted to thc AgIS wcbsite individuallv after t)roof is returncd 
from authors and belote apl)earing in an issue. 
A Consent to Publish and C«»pyright Agreement is rcquired bcfi»rc a paper will be 
published in this journal. After a Imper is a««epl'd for publication, the Providence 
office will send a Consent to Publish and C»l).vright .\greement to all authors of the 
paper. By submitting a pal)er to this journal, authors certify that the results have 
hot been submitted to nor are. they ramer consideration for publication |)v anolhcr 
journal, conference pro«eedings, or similar publication. 

Information for Authors 
Initial stlbnlission. Two copies of the pat)er should be sent directlv to the 
appropriate Editor raid the author sllould keep a coi»3". IF an edior is agreeable, 
an electronic manuscript prepared in TI)X or ECTEX nlav be submitte(l by pointing 
to an appropriate UFIL on a preprint or e-print serrer. 
The first page must consist of a descriptive title followed bv an abstract that 
summarizes the mticle in language suitable for workers in the general field (alge- 
bra, analysis, etc.). The descriptive title shoul(l be short, but informative: useless 
or vague phrases such as %orne renlarks about" or %oncerning" should be avoided. 
The abstract should be at least one conlplete sentence, and at nlost 300 words. 
Included with the footnotes to the paper should be the 201)0 Mathernoics Sbject 
Classiflcatio representing the primary and secondary subjects of the m'ticle. The 
classifications are accessible fronl . ams. org/msc/. The list of classifications is 
also available in print starting with the 1999 annual index of ll[aternaical 
views. The Mathematics SuSject Classification footnote may be followed by a list 
of ke!l words ad ptrases describing the subject matter of the article and taken 
from it. Journal abbreviations used in bibliographies are listed in the latest AIat- 
ern, a.ical Reviews ammal index. The series abbreviations are also accessible from 
.ams.org/publ±cat±ons/. To help in preparing and verifying references, the 
AMS offers MR Lookup, a Reference Tool for Linking, at . ams. org/mrlookup/. 
When the manuscript is submitted, authors should supply the editor with electronic 
addresses if available. These will be printed after the postal address at the end of 
each article. 

Electronically prepared nlanuscripts. The AgIS encourages electronica.lly 
lre ' • mlsçrlnt ,rith  et ....... çerence for ,A.A/[$-DTEX. To this end. the 



Society has prepared ,A.M$-L,TF author pa«kages for ea«h AMS publication. Au- 
thor packages include iustru«t.ious for preparing ele«trouic manuscripts, the .4MS 
Au.thoç Handbook, samples, aud a style file that geuerates the particular design 
specifica.t.ious of that publicat.iou series. Art.i«]es properly prepared using the 
X st.y]e file aud the label and ref commauds automatically enable extensive 
intra-document liukiug to the bibliography and other elements of the article for 
searchiug electrouically on the $b. Because ]iuking must ofteu be added manu- 
a]ly to electrouica]]v prepared nmuuscripts iu other forms of 
also redu«es the amouut of te«huica] interveutiou ouce the files are received by the 
AMS. This results iu fewer errors iu processiug and saves the author proofreading 
time. MS- papers also move more ecieutly through the production stream, 
helping t.o miuimize publishiug cost.s. 
MS-LAX is the highly preferred format of X. but author packages are also 
«vai]able in MS-. Those authors who make use of these style files from the 
begimfing of the writiug process will fluther reduce their own efforts. [amscripts 
pr«'pared e]e«trouicd]y iu I or p]aiu  are norma]lv uot acceptable due to 
the high amouut of teclmical time required to insure that the flic will run properly 
throgh the AMS iu-house production system. LX users will find that 
Ex is the saine as EXTX with additiomd ('Ollllil&nds to simplify the typesetting 
of matheliiatics, and sers of l)laili  should bave the foulidatiou for learniug 
Authors may ret.rieve ail author pa«kage ff'oto the AMS websie starting from 
. s. org/tex/ or via FTP to ftp. s.org (logiu  onous. enter user- 
lmme as psword, aud type cal pub/author-info). The AMS Aathor Handbook 
and the bstruction Manual are available in PDF format following the author pack- 
ages liuk from .s.org/ex/. The author package can also be obtained free 
of charge bv seudiug email to pubs.org (Interuet) or from the Publicatiou Di- 
visiou, Americau Iat.hematical Society, 201 Charles Street, Provideuce, I 02904- 
2294 USA. Wheu requesting an aut.hor packge, please speci- - or 
, Macintosh or IBI (3.5) format, and the publicatiou in which your paper will 
appear. Please be sure to include your complete mailiug address. 
At the time of submissiou, authors should iudicate if the paper has beeu prepared 
usiug MS-T&X or MS- and provide the Editor with a paper malmscript 
that matches the electrouic mamscript. The final versiou of the electronic 
uscript should be sellt to the Provideuce oce immediatelv afler the paper bas 
beeu accepged for publicatiou. The author should also seud the fiual version of the 
paper malmscript to the Editor, who will forward a copy to the Providence oce. 
Edigors will require authors to seud their electronicallv prepared mauuscripts to 
the Providence oce lu a timely fhiou. Electrouicallv prepared mauuscripts can 
be sent via email to pub-submits.org (Iuteruet) or ou diskette to the Elec- 
tronic Prepress Departmeut, American Mathematical Society, 201 Carles Street, 
Provideuce, HI 02904-2294 USA. When sending a manuscript electronicallv, plee 
be sure to include a message indicating in which publication the paper bas been 
accepted. No corrections will be accepted electronically. Authors must mark their 
changes on their proof copies and return them to the Providence oce. Complete 
instructions on how to send files are iucluded lu the author package. 

Electronic graphics. Comprehensive instructions oll preparing graphics are 
available startiug from www. oms. org/jourhtml/authors, html. A few of the nlajor 
requirements are given here. 
Subnlit files for graphics as EPS (Encapsulated PostScript) files. This iucludes 
grapllics originated via a gral)hics ap ..... r 



other COmlmter-generated images. If this is lier possible, TIFF files are acceptable 
as long as thev tan be opened in Adobe Photoshop or Illustrator. No lnatter what 
lnethod was used te produce the graphic, it is ncccssaly te provide a Imper copy te 
the AMS. 
Authors using graphics packages for the creation of elcctronic art should aise 
avoid the use of any lines thilmer than 0.5 points in wMth. lImy graphics packages 
allow the user te specify a "haMine" for a very thin line. Hairlines often look 
acceptable when proofed on a typical laser printer. However. when produced on a 
high-resolutitm laser imagesetter, hairlilms l»eCOlne nearlv invisible and will be lest 
entirelv in the final pl'inting process. 
Screens should be set te values between 15% and 85%. Screens which fall outside 
of this range are too light or too dark te print «orr,ctly. Variat tons of scrcens within 
a graphic should be n less tha.l 1(1%. 
AMS policy on making cllanges te articles after posting. Articles are 
posted te the AgIS website individually after proof is return('d frein authors and 
before appearing in an issue. Te preserve the integrity of ele«tl'mically lml»lisllcd 
articles, once an article is individually postcd te the AXlS wel»site i»ul net yct in 
ail issue, changes calmot be ma.de in place iii the Imper. llow«,ver, an "Atded aller 
posting'" section lllav be added te the Imper right l»cfore the l/efl'rences when lhere 
is a critical errer in the content of the lmp'r. The "Addcd aller posting'" sectitm 
gives the author an Ol»l»Ol'tUlfity te correct this type of critical errer lefl»re the 
article is put into ail issue for I}rilltillg and before il is then reposted with the issue. 
The "Added after posting" section relnains a permanent part of the paper. The 
AMS does net keep author-related information, such as aftïliation, current address, 
and email address, up te date after a pal»er is initially posted. 
Ollce the artMe is assigned te an issue, even if the issue has net vol been posted te 
the AMS website, corrections mav be ruade te the paper l\v submitting a traditional 
errata article te the Editor. The errata article will a.ppear in a future print issue 
and will link back and forth on the web te the original article online. 
Secure manuseript traeking on the Web alld via email. Authors Call track 
their manuscripts through the AgIS jollrnal production process using the personal 
AgIS ID and Article ID printed in the upper right-hand corner of the Consent 
te Publish ferre sent te each author who publishes in A,XIS journals. Açcess te 
the tracking system is available frein . ams. org/mstrack/ or via email sent te 
mstrack-query¢ams.org. Te access bv email, on the subject line of the message 
silnply enter the AgIS ID and Article ID. Te track lnore than one malmscript bv 
email, choose one of the Article IDs and enter the AgIS ID and the Article ID 
followed bv the word all on the subject line. An explanation of each pro<luction 
step is provided on the web through links frein the manuscript tracking screen. 
Questions tan be sent te tran-query¢ams, erg. 
TEX files available. Beginliing with the January 1992 issue of the Bttlletir and 
the 3anuary 1996 issues of Trarsactiors, Proceedings, Mathematics of Comp.uta- 
tiers, and the Jottrtal of the AMS, TEX files tan be downloaded from the AMS 
website, starting frein . a,ms. org/j ournals/. Authors without Web access may 
request their files at the address given below after the article bas been published. 
For Bulletin papers published in 1987 through 1991 and for Transactiors. Proceed- 
ings. Mathernatics of Computation, and the Journal of the AMS papers published 
in 1987 through 1995, TEX files are available upon request for authors without Web 
access by sending email to I±le-requestCams. org or by contacting the Electronic 
Prepress Department. American Mathematical Society, 201 Charles Street, Provi- 
doneo 11 fP)OOA_990 I lI A m, ....... ;hould iliclu<le the title of the paper, the 



naine(s) of the author(s), tho naine of the lmblicati«m in which tho paper h or will 
appear, and the volume and issue mlnbers if kn«»wn. The  file will be sent to 
the author making the request aller the article goe to the printer. If the requestor 
«an receive Interner email, please include the email address to which the file should 
be sent. Otherwise please indicate a diskette [brmat and postal address to which 
a disk should be mailed. Note: Because  production at the AMS soinetimes 
requires extra fonts and macros that are not ver publicly a»-ailable,  files cannot 
be guaranteed to run through the author's version of  without errors. The AMS 
regrets that it cann»t provide support to eliminate snch <,rrors in the author's T 
environment. 
Inquiries. Ara inquiries concerning a pal»er that has been accepted for pub- 
lication that calmot be a.nswered via the malmscript tracking system lnentioned 
above should l>e sent to tr-query+s, org or directlv to the Electronic Prepress 
Departmenl, Ameri«an lath,mati('al So«iety, 201 Charles Stl'eet. Providence. I 
02904-2294 USA. 



Editors 
The traditional method of sul)mitting a paper is te send two hard copies te the appro- 
priate editor. Suhjects, and the editors associated with them, are listed below. 
In principle the Transactions welcomes electronic suhmissions, and seine of t he editors, 
those whose nallles appear 1)elow with an aslerisk (*). bave indicated that they prefer 
them. Editors reserve the right te request hard ce»pies after papers have been sul»nlitted 
electronically. Aut hors are advised te make preliminary inquiries te editors as te whether 
they are likely te 1)e al)le te handle submissions iu a particular electronic ferre. 
Algebra and algebraic geometry. KAREN E. SMIT|I. Department of Mathematics, Uni- 
versity of Michigan, Ann Arbor, iMI 4,119-1109 USA: e-mail: k«smith@umich.«du 
Algebraic geometry. DAN ABRAM()VI('I[, Deparlulent of Malhematics, Boston Univer- 
sity. 111 Cummington Street, Boston, MA 11221.5 USA: e-mail: abramovicbu.edu 
Algebraic topology and cohomology of groups. STE\VAHT PIIDDY, Deartment of 
Mathematics, Northwestern University, 2033 Sheridan Road, Evanston. IL 60208-2730 L'SA; 
e-mail: priddy@math, nuu. 
* Cornbinatorics. SERGEY FOMIN, Department of Mathematics, Èast Hall. ['niversity of 
Michigan, Ann Arbor. MI 48109-1109 USA; e-mail: fominumich. «du 
Complex analysis and geometry. D. H. PHONG. Departmen! of Mathematics. ('ohmlbia 
University. 2990 Broadway. New h ork, NY 1127-002.q l'SA: e-mail: phongma'ch, coumbia. «du 
* Differential geonmtry and global analysis. LISA ('..IEFFI|"Y. 1)epartm('nt of Malh- 
ematics, University of Toronto. 100 St. (;eorge Strcet. T,»ronto, ()ntario, Canada lklSS 3(13; 
e-mail: jaffray@math. 1;oron'co. adu 
Dynamical systems and ergodic theory. I¢OBEICT F. \VII,LIAM, Department of Math- 
ematics, University of Texas, Austin, TX 78712-102 USA: e-mail: bobOmath.utexas, edu 
* Geometric analysis, TOBIAS COLDING. Courant Institute. New hork University, 251 
Mercer Street, New York, NY 10012 USA: e-mail: coldingOcis.nyu.edu 
Geornetric topology, knot theory, and hyperbolic geolnetry, ABIGAIL TtlOiMP- 
SON. Department of Mat.hematics, University of California. Davis. CA 95616-5224 ['SA: e-mail: 
/hompson@ma/h. ucdavis, edu 
Harmonic analysis, ALEANDER NAGEL. Department of Mathcnlatics, University of Wis- 
consin, 480 Lincoln Drive, Ikladison. \VI 53706-1313 USA; e-nail: nagel@ma*ch. isc.edu 
Harmonic analysis, representation theory, and Lie theory. IqOBEHT .I. STANTON, 
Department of Mathematics, Ohio State [rniversity. 231 Vest ltqth Avenue, Columbus. OH 43210- 
1174 USA: e-maih stantonmath, ohio-s/a/e, edu 
* Logic. THEODORE SLANIAN, Department of Mathematics. University of Clifornia. Berke- 
ley. CA 94720-3840 USA: e-maih slamanmath.berk«l«y.«du 
Nurnber theory. HAROLD G. DIAMOND, Department of lklathematics, Universitv of llli- 
nois, 1409 WesI Green Street. Urbana, IL 61801-2917 USA; e-mail: diamondmath.uiuc 
* Ordinary differential equations, partial differential equations, and applied math- 
ematics, PETER W. BATES, Department of Mathematics, Midfigan State University, Èast 
Lansing, lklI 48824-1027 USA: e-maih bat«smath.msu.«du 
* Partial differential equations. PATF{I('IA E. BAUMAN, Department of Mathematics, 
Purdue University, West Lafayette. IN 47907-1395 t'SA; e-mail: baumanmath.purdu« 
* Probability and statistics. K RZYSZTOF BURDZY. Department of Mat hemat ics, Univer- 
sity of Washington, Box 354350. Seattle, %VA 98195-4350 USA; e-mail: burdzy@ath. ashington. 
* Real analysis and partial differential equations, DANIEL TATARU, Department of 
Mathematics, University of California. Berkeley, CA 94720 USA: e-mail: tatarumath 
All other communications to the editors should be addressed o the lklanaging Editor, 
WILLIAM BECKNER, Department of Mathernatics, University of Texas, Austin, TX 
78712-1082 USA: e-mail: beckner@,ath.utexas, edu 

MEMOIRS OF THE AMERICAN MATHEMATICAL SOCIETY 
Memoirs is devoted te research in pure and applied mathenmtics of the saine nature as 
Transactions. An issue consists of one or more separately bound research tracts for which 
the authors provide reproduction copy. Papers intended for Mernoirs should normally be 
at least 80 pages in length. Memoirs }las the saine editorial committee as Transactions: 
so such DaDers shofld t)e add,-o,:,,¢l t ..... r the editors listed al)ove. 



TNSACTIONS 

AMERICAN MATHEMATICAL SOCIETY 

TANSACTIONS " 

I Journal overview: This jour- 
nal is devoted to research arti- 
cles in ail areas of pure and 
applied mathematics.To be pub- 
lished in the Transactions, a 
paper must be correct, new, 

and significant. Further, it must 
-«,«,,,,, be well written and of interest 
.  ,,', ............ to a substal number of 
information: I   
.'"J°r?'°":X I ' TRANSACTIONS ., 
inloratlon an  ]   o»t .  
befundnme  -  " 
Submission page. I  , 

AZDED 

The AMS has released enhanced versions of its electronic journals. 
These upgrades improve usefulness and relevante for both journal 
subscribers and journal authors. 

A 30-day free trial is available to corporations and institutions. 
A downloadable Free Trial Form is available at: www.ams.org/ 
customers/ejournaltrial.pdf. 
Contact AMS Membership and Customer Services, 201 Charles Street, Providence, RI 
02904-2294, USA; phone 1-800-321-4267 or 1-401-455-4000 worldwide; 
fax 1-401-455-4046; email: cust-serv@ams.org. Note:A signed license agreement is 
required for AMS electronic iournal subscriptions.A newly updated and expanded 
agreement tan be found online at http:llwww.ams.org/customers/jour-license.html. 



TRANSACTIONS OF THE AMERICAN MATHEMATICAL SOCIETY 
CONTENTS 
Vol. 355, No. 7 Whole No. 818 July 2003 

Borislav Karaivanov, Pencho Petrushev, and Robert C. Sharpley, 
Algorithlns fi)r nonlinear piecewise polynolnial at)t)roximation: Theoret- 
i('al aspects ......................................................... 2585 
J6rg Brendle, The ahnost-(lisjointness mnni)er may have countable 
('otimdity ........................................................ 2633 
Alina Carmen Cojocaru, Cyclicity of Chi ellil)tic «urves niodulo p 2651 
Tonghai Yang, Ta.vlor expansion of ail Eiscnstein serics .................. 2663 
Eric Freenlan, Systenis of diagonal Diol)hantine ilwqualities ............. 2675 
Francisco Javier Gallego and Bangere P. Purnaprajna, On the 
ean()ni('al rings ()f ('overs ()t" sm'faces of minimal degree ............... 2715 
H. H. Brungs and N. I. Dubrovin, A classification and eXalliples of rank 
on(, chain «lomaius ................................................... 2733 
Donald W. Barnes, ()n the Sl)e('tral sequelwe ('onstru«tors of Guichardet 
and Stefan ....................................................... 2755 
Steven Lillywhite, Folmalitv in an equivariant sctting ................... 2771 
Neil Hindlnan, Dona Strauss, and Yevhen Zelenyuk, Large re('tangular 
semigroul)S in St()lie-Cech ('onq)a('tifica.tions .......................... 2795 
Takehiko Yamanouchi, Galois groul)S of quantuui group actions and 
regularity of fix('d-l)oint algel)ras ..................................... 2813 
Boo Rira Choe, Hyungwoon Koo, and Wayne Smith, Çoniposition 
Ol)('lators a('ting on holoniorphi(" Sol)olev spaces ..................... 2829 
B. Jakubczyk and M. Zhitomirskii, Distributions of corank 1 and their 
characteristic vector fiel(|s ........................................... 2857 
E. Boeckx, Wlien are the tangent Sl)here i)undles of a lielnalmian manifold 
re(hicil)le'? ........................................................... 2885 
Henri Colnman, Criteria for large deviations ............................ 2905 
Seung Jun Chang, Jae Gil Choi, and David Skoug, Integration by parts 
fonnulas involving gelleralizcd Fourier-Feymnan trallsfornls on flill('tioll 
sl)ace ................................................................ 2925 
Michiko Yuri, Thermodynamic formalisni for countable to one iklarkov 
systelnS ....................................